ddlx 4 Posted May 3, 2013 Report Share Posted May 3, 2013 Hi all I would like to scrappe the google serp urls. This code from 'search result scrapper' (by Kreatus) was working fine.But it not working now (i think google have change his page) if($comparison(#current total, "<", #total results)) { then { if($search page("onmousedown=\"return rwt(this")) { then { add list to list(%result urls, $scrape attribute(<outerhtml=w"<a href=\"*\" class=\"*\" onmousedown=\"return rwt(this*)\">*</a>">, "href"), "Delete", "Global") } else { add list to list(%result urls, $page scrape("<a href=\"/url?q=", "&sa="), "Delete", "Global") } } click(<style="background-position:-96px 0;width:71px">, "Left Click", "No") wait for browser event("DOM Ready", "") } else { } } I think the wrong code is there :$scrape attribute(<outerhtml=w"<a href=\"*\" class=\"*\" onmousedown=\"return rwt(this*)\">*</a>">, "href") Is someone got the solution ? Quote Link to post Share on other sites
Walter1 8 Posted May 6, 2013 Report Share Posted May 6, 2013 Same problem here, seems they dont have that specific class anymore. Quote Link to post Share on other sites
Kreatus (Ubot Ninja) 422 Posted May 6, 2013 Report Share Posted May 6, 2013 This should work. clear list(%result urls)clear list(%final url)if($comparison(#current total, "<", #total results)) { then { if($search page("onmousedown=\"return rwt(this")) { then { add list to list(%result urls, $scrape attribute(<tagname="h3">, "outerhtml"), "Delete", "Global") } else { add list to list(%result urls, $page scrape("<a href=\"/url?q=", "&sa="), "Delete", "Global") } } click(<style="background-position:-96px 0;width:71px">, "Left Click", "No") wait for browser event("DOM Ready", "") } else { }}loop($list total(%result urls)) { add item to list(%final url, $find regular expression($next list item(%result urls), "(?<=href=\").*?(?=\" onmousedown)"), "Delete", "Global")}load html($replace($text from list(%final url, ""), "", "<br>")) 1 Quote Link to post Share on other sites
ddlx 4 Posted May 7, 2013 Author Report Share Posted May 7, 2013 Cool, i ll try to insert this code in my boot... Will tel you if it work fine. Quote Link to post Share on other sites
ddlx 4 Posted May 8, 2013 Author Report Share Posted May 8, 2013 Hi, i am back There is somethink wrong in : loop($list total(%result urls)) { add item to list(%final url, $find regular expression($next list item(%result urls), "(?<=href=\").*?(?=\" onmousedown)"), "Delete", "Global")} It does not correctly find the "http... etc" Quote Link to post Share on other sites
ddlx 4 Posted May 8, 2013 Author Report Share Posted May 8, 2013 Hum, i was wrong, it's working... ... strange computer ... Quote Link to post Share on other sites
iceman007 0 Posted May 10, 2013 Report Share Posted May 10, 2013 Only fairly new to Ubot so can someone explain the code above if it is working for some people as there appears to be no navigation module so there must be something missing? Quote Link to post Share on other sites
ddlx 4 Posted May 10, 2013 Author Report Share Posted May 10, 2013 Arf lol, this is a partial code to scrape google results. I ll post the complète code when finished my bot Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.