PRO 69 Posted September 5, 2021 Report Share Posted September 5, 2021 Hello community, Because many people have asked for help regarding this, here is some ready-made UBot code for you. This example code works with UBot Studio only (without the need for any plugins being installed). For more demanding scenarios there are other, more sophisticated ways to achieve things by utilizing additional plugins for UBot Studio. Feel free to leave a like if this helped you out. Last updated 2021-09-06 13:55 CET navigate("https://www.google.com","Wait") wait(2) comment("Click Google Privacy Consent Button if present") click(<class="tHlp8d">,"Left Click","No") wait(1) comment("Do Search") navigate("https://www.google.com","Wait") wait(1) change attribute(<name="q">,"value","weather") type text(<name="q">,"\{ENTER\}","Standard") comment("Wait for Results Page to Load") wait for element(<id="result-stats">,"","Appear") wait(1) comment("Extract Organic Results HTML elements") set(#scrape_organic_results_html,$scrape attribute(<(tagname="div" AND class="yuRUbf")>,"innerhtml"),"Global") comment("Filter Organic Results HTML elements") set(#scrape_organic_results_html,$find regular expression(#scrape_organic_results_html,"<a href=\".*?\\\""),"Global") set(#scrape_organic_results_html,$replace(#scrape_organic_results_html,"<a href=\"#\"",""),"Global") set(#scrape_organic_results_html,$replace(#scrape_organic_results_html,"<a href=\"",""),"Global") set(#scrape_organic_results_html,$replace(#scrape_organic_results_html,"\"",""),"Global") comment("Put extracted Links into List") clear list(%organic_results_href) add list to list(%organic_results_href,$list from text(#scrape_organic_results_html,$new line),"Don\'t Delete","Global") comment("Output Results") alert("You scraped {$list total(%organic_results_href)} organic results! {%organic_results_href}") 1 1 Quote Link to post Share on other sites
PRO 69 Posted September 5, 2021 Author Report Share Posted September 5, 2021 If you prefer using ExBrowser, you can use the following XPath expression instead in order to extract the organic results: x://div[contains(@class,"yuRUbf")]/a[1] Use the XPath expression inside a "scrape list element attribute" function and set the attribute to be "href". Congratulations 🤖 Quote Link to post Share on other sites
Urgent Enquirer 0 Posted September 6, 2021 Report Share Posted September 6, 2021 16 hours ago, PRO said: Hello community, Because many people have asked for help regarding this, here is some ready-made UBot code for you. This example code works with UBot Studio only (without the need for any plugins being installed). For more demanding scenarios there are other, more sophisticated ways to achieve things by utilizing additional plugins for UBot Studio. Feel free to leave a like if this helped you out. Last updated 2021-09-05 20:16 CET navigate("https://www.google.com","Wait") wait(3) comment("Click Google Consent Button if present") click(<class="tHlp8d">,"Left Click","No") wait(1) comment("Do Search") navigate("https://www.google.com","Wait") change attribute(<name="q">,"value","weather") type text(<name="q">,"\{ENTER\}","Standard") comment("Wait for Results Page to Load") wait for element(<id="result-stats">,"","Appear") wait(1) comment("Extract Organic Results HTML elements") set(#scrape_organic_results_html,$scrape attribute(<(tagname="div" AND class="yuRUbf")>,"innerhtml"),"Global") comment("Filter Organic Results HTML elements") set(#scrape_organic_results_html,$find regular expression(#scrape_organic_results_html,"<a href=\".*?\\\""),"Global") set(#scrape_organic_results_html,$replace(#scrape_organic_results_html,"<a href=\"#\"",""),"Global") set(#scrape_organic_results_html,$replace(#scrape_organic_results_html,"<a href=\"",""),"Global") set(#scrape_organic_results_html,$replace(#scrape_organic_results_html,"\"",""),"Global") comment("Put extracted Links into List") clear list(%organic_results_href) add list to list(%organic_results_href,$list from text(#scrape_organic_results_html,$new line),"Don\'t Delete","Global") comment("Output Results") alert("You scraped {$list total(%organic_results_href)} organic results! {%organic_results_href}") Is working! But care to explain those lines one by one which you added extra to beat the google javascript mechanism that foils bots ? Care to explain what the regex does ? And why you did all the replacings ? Talking about these lines: set(#scrape_organic_results_html,$scrape attribute(<(tagname="div" AND class="yuRUbf")>,"innerhtml"),"Global") comment("Filter Organic Results HTML elements") set(#scrape_organic_results_html,$find regular expression(#scrape_organic_results_html,"<a href=\".*?\\\""),"Global") set(#scrape_organic_results_html,$replace(#scrape_organic_results_html,"<a href=\"#\"",""),"Global") set(#scrape_organic_results_html,$replace(#scrape_organic_results_html,"<a href=\"",""),"Global") set(#scrape_organic_results_html,$replace(#scrape_organic_results_html,"\"",""),"Global") Quote Link to post Share on other sites
Urgent Enquirer 0 Posted September 6, 2021 Report Share Posted September 6, 2021 @Pro How come on my end, the gogole search result link attributes are shown as: <tagname="h3"> <class="LC20lb DKV0Md"> While on your end: <(tagname="div" AND class="yuRUbf")> Ubot shows our classes and tagnames are different. Why ? Your code is working on my end, however. But remember, my Ubot doesn't show tagname or class values the same as your's. Quote Link to post Share on other sites
PRO 69 Posted September 6, 2021 Author Report Share Posted September 6, 2021 @Urgent Enquirer, glad you found it working just fine 👍 You ask: "How come on my end, the google search result link attributes are shown as XY while on your end AB?" This is because of skill. For me it looks the same and it doesn't just "show". Needs your own skill to figure out the correct selectors. And because the search result page is difficult to figure out, I put this example code to make it easy, showing you a ready-made solution. Please understand that I won't explain every line of code one by one. Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.