Jump to content
UBot Underground

Ubot Scrapes Irrelevant Links


Recommended Posts

 

Hi,

I am trying to scrape the search result links from this page:

https://www.google.com/search?q=covid+vaccine

Snippet:

add list to list(%urls,$scrape attribute(<outerhtml=w"<h3 class=\"LC20lb DKV0Md\">*</h3>">,"fullhref"),"Delete","Global")

Note the above code is totally unrelated to the SERP page links (page 1, page 2, page 3 and so on). So if it is unrelated to such SERP links then why is Ubot scraping these SERP links ?

Wrongfully is scraping the Search Result Page links (page 1, page 2, page 3 and so on) that you see at the bottom of the Google search result page).
Test the following code out and see for yourselves.
 
 

Code Context:

ui text box("Keywords",#ui textbox_keywords)
ui drop down("Max Results","10,20,30,40,50,60,70,80,90,100",#ui dropdown_max results)
ui save file("Save To",#ui save file_save to)
ui stat monitor("Total Results",$list total(%urls))
divider
divider
set(#max results,#ui dropdown_max results,"Global")
set(#keywords,#ui textbox_keywords,"Global")
set(#save to,#ui save file_save to,"Global")
divider
navigate("https://www.google.com/search?q={#keywords}","Wait")
wait($rand(1,5))
clear list(%urls)
loop while($both($comparison($list total(%urls),"<= Less than or equal to",#max results),$either($exists(<id="pnnext">),$exists(<outerhtml="<span style=\"display:block;margin-left:53px\">Next</span>">)))) {
    add list to list(%urls,$scrape attribute(<outerhtml=w"<h3 class=\"LC20lb DKV0Md\">*</h3>">,"fullhref"),"Delete","Global")
    if($exists(<id="pnnext">)) {
        then {
            comment("Click for \"Next\" Result.")
            click(<id="pnnext">,"Left Click","No")
        }
        else {
            if($exists(<outerhtml="<span style=\"display:block;margin-left:53px\">Next</span>">)) {
                then {
                    comment("Click for \"Next\" Result.")
                    click(<outerhtml="<span style=\"display:block;margin-left:53px\">Next</span>">,"Left Click","No")
                }
                else {
                    alert("ERROR ALERTt: 

Cannot detect \"Link to NEXT Page\"!
Quitting the session.")
                    divider
                }
            }
            divider
        }
    }
    divider
    wait for browser event("Page Loaded","")
}
save to file(#save to,%urls)
 
 
Edited by Urgent Enquirer
Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...