cervant41_ 0 Posted December 2, 2009 Report Share Posted December 2, 2009 Ya know what's funny, i probably have roughly 15 different bots in all stages of development (ubot a.d.d? lol) and i've come across something so stupidly simple, i feel dumb. can someone post up a quick workflow for scraping only the urls of a google search. Dumb, so dumb... Quote Link to post Share on other sites
Gogetta 263 Posted December 2, 2009 Report Share Posted December 2, 2009 Something like this. nav ( google)--> choose by attribute-> add to list-->scrape chosen attribute ( href)--->save to file (and put your list that you used). Thats a way to do it. Edit :oops, I had it wrong so... Quote Link to post Share on other sites
cardine 0 Posted December 2, 2009 Report Share Posted December 2, 2009 You should probably also include an if statement that doesn't add the url if it contains 'google.com' in it. That way it will only add links that leave google (search results). Quote Link to post Share on other sites
some_guy 11 Posted December 3, 2009 Report Share Posted December 3, 2009 Are you wanting all URLs in the 1000 results allowed or just the first page? If just the first page "should" be easy. If all pages then would need to scrape pagination links first into a list, then loop through the pagination list, calling a scrape current page method/block/subprogram, whatever it is called in Ubot. Actually have been meaning to get on with Ubot, am going to give this a go, will post if make any progress with i. Quote Link to post Share on other sites
msimurin 4 Posted December 13, 2009 Report Share Posted December 13, 2009 This is easy, i could explain it but better check seths tutorial on keyword google tool, you can find it on vimeo.com Its the part when he starts scraping data to check competition, just watch that part and you will understand. Just a hint: use $page scrape on <cite> tag as this is google html tag for links, its really simple to do. You could also go to advanced search options and use link format for nav that shows 100 results, it will go much faster if you do this. Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.