Jump to content
UBot Underground

How To Extract Organic Search Results from Google with UBot [September 2021]


Recommended Posts

Hello community,

Because many people have asked for help regarding this, here is some ready-made UBot code for you.

This example code works with UBot Studio only (without the need for any plugins being installed). For more demanding scenarios there are other, more sophisticated ways to achieve things by utilizing additional plugins for UBot Studio.  

Feel free to leave a like if this helped you out.

Last updated 2021-09-06 13:55 CET

navigate("https://www.google.com","Wait")
wait(2)
comment("Click Google Privacy Consent Button if present")
click(<class="tHlp8d">,"Left Click","No")
wait(1)
comment("Do Search")
navigate("https://www.google.com","Wait")
wait(1)
change attribute(<name="q">,"value","weather")
type text(<name="q">,"\{ENTER\}","Standard")
comment("Wait for Results Page to Load")
wait for element(<id="result-stats">,"","Appear")
wait(1)
comment("Extract Organic Results HTML elements")
set(#scrape_organic_results_html,$scrape attribute(<(tagname="div" AND class="yuRUbf")>,"innerhtml"),"Global")
comment("Filter Organic Results HTML elements")
set(#scrape_organic_results_html,$find regular expression(#scrape_organic_results_html,"<a href=\".*?\\\""),"Global")
set(#scrape_organic_results_html,$replace(#scrape_organic_results_html,"<a href=\"#\"",""),"Global")
set(#scrape_organic_results_html,$replace(#scrape_organic_results_html,"<a href=\"",""),"Global")
set(#scrape_organic_results_html,$replace(#scrape_organic_results_html,"\"",""),"Global")
comment("Put extracted Links into List")
clear list(%organic_results_href)
add list to list(%organic_results_href,$list from text(#scrape_organic_results_html,$new line),"Don\'t Delete","Global")
comment("Output Results")
alert("You scraped {$list total(%organic_results_href)} organic results!

{%organic_results_href}")

  • Like 1
  • Thanks 1
Link to post
Share on other sites

If you prefer using ExBrowser, you can use the following XPath expression instead in order to extract the organic results:

x://div[contains(@class,"yuRUbf")]/a[1]

Use the XPath expression inside a "scrape list element attribute" function and set the attribute to be "href".

Congratulations 🤖

Link to post
Share on other sites
16 hours ago, PRO said:

Hello community,

Because many people have asked for help regarding this, here is some ready-made UBot code for you.

This example code works with UBot Studio only (without the need for any plugins being installed). For more demanding scenarios there are other, more sophisticated ways to achieve things by utilizing additional plugins for UBot Studio.  

Feel free to leave a like if this helped you out.

Last updated 2021-09-05 20:16 CET

navigate("https://www.google.com","Wait")
wait(3)
comment("Click Google Consent Button if present")
click(<class="tHlp8d">,"Left Click","No")
wait(1)
comment("Do Search")
navigate("https://www.google.com","Wait")
change attribute(<name="q">,"value","weather")
type text(<name="q">,"\{ENTER\}","Standard")
comment("Wait for Results Page to Load")
wait for element(<id="result-stats">,"","Appear")
wait(1)
comment("Extract Organic Results HTML elements")
set(#scrape_organic_results_html,$scrape attribute(<(tagname="div" AND class="yuRUbf")>,"innerhtml"),"Global")
comment("Filter Organic Results HTML elements")
set(#scrape_organic_results_html,$find regular expression(#scrape_organic_results_html,"<a href=\".*?\\\""),"Global")
set(#scrape_organic_results_html,$replace(#scrape_organic_results_html,"<a href=\"#\"",""),"Global")
set(#scrape_organic_results_html,$replace(#scrape_organic_results_html,"<a href=\"",""),"Global")
set(#scrape_organic_results_html,$replace(#scrape_organic_results_html,"\"",""),"Global")
comment("Put extracted Links into List")
clear list(%organic_results_href)
add list to list(%organic_results_href,$list from text(#scrape_organic_results_html,$new line),"Don\'t Delete","Global")
comment("Output Results")
alert("You scraped {$list total(%organic_results_href)} organic results!

{%organic_results_href}")

Is working!

But care to explain those lines one by one which you added extra to beat the google javascript mechanism that foils bots ? Care to explain what the regex does ? And why you did all the replacings ?

Talking about these lines:

set(#scrape_organic_results_html,$scrape attribute(<(tagname="div" AND class="yuRUbf")>,"innerhtml"),"Global")
comment("Filter Organic Results HTML elements")
set(#scrape_organic_results_html,$find regular expression(#scrape_organic_results_html,"<a href=\".*?\\\""),"Global")
set(#scrape_organic_results_html,$replace(#scrape_organic_results_html,"<a href=\"#\"",""),"Global")
set(#scrape_organic_results_html,$replace(#scrape_organic_results_html,"<a href=\"",""),"Global")
set(#scrape_organic_results_html,$replace(#scrape_organic_results_html,"\"",""),"Global")

Link to post
Share on other sites

@Pro

 

How come on my end, the gogole search result link attributes are shown as:

<tagname="h3">

<class="LC20lb DKV0Md">

 

While on your end:

<(tagname="div" AND class="yuRUbf")>

 

Ubot shows our classes and tagnames are different. Why ?

Your code is working on my end, however. But remember, my Ubot doesn't show tagname or class values the same as your's.

Link to post
Share on other sites

@Urgent Enquirer, glad you found it working just fine 👍

You ask: "How come on my end, the google search result link attributes are shown as XY while on your end AB?"

This is because of skill. For me it looks the same and it doesn't just "show". Needs your own skill to figure out the correct selectors.

And because the search result page is difficult to figure out, I put this example code to make it easy, showing you a ready-made solution.

Please understand that I won't explain every line of code one by one.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...