I'm having a brain fart..

[cervant41_ 0]

Ya know what's funny, i probably have roughly 15 different bots in all stages of development (ubot a.d.d? lol) and i've come across something so stupidly simple, i feel dumb.

 

can someone post up a quick workflow for scraping only the urls of a google search. Dumb, so dumb...

[Gogetta 263]

Something like this.

 

nav ( google)--> choose by attribute-> add to list-->scrape chosen attribute ( href)--->save to file (and put your list that you used).

 

Thats a way to do it.

 

Edit :oops, I had it wrong so...

[cardine 0]

You should probably also include an if statement that doesn't add the url if it contains 'google.com' in it. That way it will only add links that leave google (search results).

[some_guy 11]

Are you wanting all URLs in the 1000 results allowed or just the first page?

 

If just the first page "should" be easy.

 

If all pages then would need to scrape pagination links first into a list, then loop through the pagination list, calling a scrape current page method/block/subprogram, whatever it is called in Ubot.

 

Actually have been meaning to get on with Ubot, am going to give this a go, will post if make any progress with i.

 

 

[msimurin 4]

This is easy, i could explain it but better check seths tutorial on keyword google tool, you can find it on vimeo.com

 

Its the part when he starts scraping data to check competition, just watch that part and you will understand.

 

Just a hint: use $page scrape on <cite> tag as this is google html tag for links, its really simple to do.

 

You could also go to advanced search options and use link format for nav that shows 100 results, it will go much faster if you do this.