Josh 37 Posted October 14, 2012 Report Share Posted October 14, 2012 I really loved the huge list of footprints that TJ and Willy Wonka provided the community. My hat is off to you two. So I wanted to do something with ubot to give to the community to help with searching footprints. I'm working on this bot that allows you to import a list of footprints and a list of keywords. It all works perfectly fine except for one part. Scraping the urls from the Google SERP. It will scrape the urls sometimes and not others. I have gone through the forums and tried all the scraping tricks I have seen and none of them seem to work consistently. This is the last part needed for this bot and then I will post it up for free to the community. If anyone has any insight on this it would help out a lot. I have attached the bot! footprint-website-finder.ubot Quote Link to post Share on other sites
k1lv9h 76 Posted October 15, 2012 Report Share Posted October 15, 2012 Hi, I found a small problem.This bot was skipping the last page of result(s). The last result(s) page is missing "<innertext="Next">". I changed the loop while that is looking for "<innertext="Next:>". This allowed the last page of result(s) to be processed. Updated code:footprint-website-finder-001.ubot Kevin 1 Quote Link to post Share on other sites
Josh 37 Posted October 15, 2012 Author Report Share Posted October 15, 2012 Ah!!!! Thanks Kevin. I didn't catch that it was doing that. I also needed to add an if command for if the keyword list is empty. I'll be uploading it to the buy/sell bots section in a moment. Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.