steelersfan 38 Posted July 25, 2016 Report Share Posted July 25, 2016 I am trying to make a bot that scrapes bing search. The problem I am having, is that when the search engine runs out of pages in the index, the bot keeps going until the number of loops is used up. I want the bot to stop and return a message when the index no longer has pages in it. With google, there is a message at the last page (most of the time), so that works to stop a google scrape, but bing has no such message. How can I get the bot to understand that there are no more pages in the index to search? As it is now, the bot just clicks the next button in a loop until the search item is found. This is for all of the search engines that I am scraping with it. So how can I make sure the bot understands that thee is no longer a "next" button? Quote Link to post Share on other sites
HelloInsomnia 1103 Posted July 25, 2016 Report Share Posted July 25, 2016 You would set a true/false variable then loop based on that variable. If there is a next button the variable stays the same, and if there is no next button then change the variable like a sort of on off switch so the loop can stop. Here is a basic example you will need to add in the scraping part and waiting for page to load, etc. set(#running,"true","Global") loop while($comparison(#running,"=","true")) { comment("Scrape page here") divider if($exists(<title="Next page">)) { then { comment("If there is a next button click on it") click(<title="Next page">,"Left Click","No") } else { comment("If there is no next button then set running to be false so that it will exit the loop") set(#running,"false","Global") } } } 1 Quote Link to post Share on other sites
steelersfan 38 Posted July 26, 2016 Author Report Share Posted July 26, 2016 Thank you HelloInsomnia! I was using various methods of determining if the text was on the page, but it was not accurate and hardly worked. I had wondered if I could indeed use the title or id, etc. Now I know how to! Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.