Stopping A Bot Loop On Bing

[steelersfan 38]

I am trying to make a bot that scrapes bing search. The problem I am having, is that when the search engine runs out of pages in the index, the bot keeps going until the number of loops is used up. I want the bot to stop and return a message when the index no longer has pages in it. With google, there is a message at the last page (most of the time), so that works to stop a google scrape, but bing has no such message.

 

How can I get the bot to understand that there are no more pages in the index to search? As it is now, the bot just clicks the next button in a loop until the search item is found. This is for all of the search engines that I am scraping with it. So how can I make sure the bot understands that thee is no longer a "next" button?

[HelloInsomnia 1103]

You would set a true/false variable then loop based on that variable. If there is a next button the variable stays the same, and if there is no next button then change the variable like a sort of on off switch so the loop can stop.

 

Here is a basic example you will need to add in the scraping part and waiting for page to load, etc.

set(#running,"true","Global")
loop while($comparison(#running,"=","true")) {
    comment("Scrape page here")
    divider
    if($exists(<title="Next page">)) {
        then {
            comment("If there is a next button click on it")
            click(<title="Next page">,"Left Click","No")
        }
        else {
            comment("If there is no next button then set running to be false
so that it will exit the loop")
            set(#running,"false","Global")
        }
    }
}

[steelersfan 38]

Thank you HelloInsomnia!

 

I was using various methods of determining if the text was on the page, but it was not accurate and hardly worked. I had wondered if I could indeed use the title or id, etc. Now I know how to!