Scraping Google Trends RSS Feed

[scriptbit 0]

I got Google Trends RSS going to my Google Reader. Every hour there is an update. I want to scrape all the keywords and put those in a database on my PC so later I can create a bot that matches that database against another keyword database (which I have not decided what is best to use, spreadsheet or actual database).

 

So if you were at the same starting point as I, per this screenshot, how would you guys begin to scrape, save, and what database to use - spreadsheet, mysql, google spreadsheets? I am going to screenshot my progress on this thread.

 

http://i56.tinypic.com/6qitky.jpg

[JohnB 255]

I tried this yesterday but couldn't find my way to that page...let me know how to get that far and I can tell you the rest of what you need to know.

 

John

[scriptbit 0]

I tried this yesterday but couldn't find my way to that page...let me know how to get that far and I can tell you the rest of what you need to know.

 

John

 

Here's how I got to that page. You will need gmail first tho.

 

1. Go to http://www.google.com/trends

2. click "More Hot Searches »" link. It will take you to here http://www.google.com/trends/hottrends?sa=X

3. The rss feed link is on that page. I clicked it (http://www.google.com/trends/hottrends/atom/hourly)

4. login to gmail, click "reader" up at the top of your gmail.

5. once in reader, click "Add a subcription", paste the rss feed link in there and add it. Now every hour it will update with new trends.

 

Now you are at that screenshot page from above.

[JohnB 255]

Ok, I'll run it

[scriptbit 0]

Ok, I'll run it

 

I await your answer ANXIOUSLY lol. Thanks.

[Praney Behl 314]

Is it the title you're trying to scrape? If you can please clarify what exactly are you trying to scrape that will good.

[scriptbit 0]

Is it the title you're trying to scrape? If you can please clarify what exactly are you trying to scrape that will good.

 

Ok I want to:

 

1. open up each update, scheduled every hour. So hourly must perform this task. How to do that?

2. scrape the keyword list contents > save to text file (or mysql database) How to do that?

3. each keyword in the list is hyperlinked. when you click it, it takes you to page with related keywords/blogs/news sites. All I want scraped > save to text file (or mysql database) How to do that?

[JohnB 255]

Not sure if this is what you're going for...

 

trends.ubot

 

 

John

[scriptbit 0]

Not sure if this is what you're going for...

 

trends.ubot

 

 

John

 

Thank you very much! I ran your bot. I attached my results. It seems close to what I want to do.

 

However, <H2 class=entry-title>*</H2><SPAN class=entry-secondary-snippet>- <SPAN class=snippet>*</SPAN></SPAN>

 

Can u tell me why this was chosen to be search string?

 

This is what I want to do:

 

1. open up each update, scheduled every hour. So hourly must perform this task. How to do that?

2. scrape the keyword list contents > save to text file (or mysql database) How to do that?

3. each keyword in the list is hyperlinked. when you click it, it takes you to page with related keywords/blogs/news sites. All I want scraped > save to text file (or mysql database) How to do that?

 

Ultimately, I think if I could start storing these keywords in one location, then start storing cpa offers in another list, then I could match an emerging trend with a cpa offer. So, i could use this as a "tell" system, telling me what I should be targeting. Only thing is i need a way to send to myself some kind of report telling me what i should promote based on these trends. Thoughts?

trends.txt