mjmccoy 0 Posted June 29, 2011 Report Share Posted June 29, 2011 I am creating a bot to scrape a local listings website. The way I have it set up is. I have used the UI to enter in a search term example " pizza atlanta " the bot then navigates to the site enters the search text and clicks the search button The site then produces a list of results it also creates multiple pages or results for this example it came back with 200 listings 20 per pages. I have created the first scraper which will grab the urls on the first page. But having trouble figuring out how to get the bot to move to the next page and next until it has grabbed all the urls for that search result. Is this a loop command. I was trying to get a loop to press the next button. But I am not sure how it will recognize the amount of pages and just keep going and never stop? Any Ideas? the next state I want the bot to go to the urls 1 by 1 and scrape pieces of content to add to a CSV but I will need to finish this part first. Please let me know M Quote Link to post Share on other sites
UBotBuddy 331 Posted June 29, 2011 Report Share Posted June 29, 2011 Can you post your bot? There are many possible ways to accomplish this. Quote Link to post Share on other sites
mjmccoy 0 Posted June 29, 2011 Author Report Share Posted June 29, 2011 Can you post your bot? There are many possible ways to accomplish this. Here is the bot I have played around a little so I am not sure if it all still works correctly because I was trying to add loop and got hung up.manta scrapper.ubot Quote Link to post Share on other sites
LoWrIdErTJ - BotGuru 904 Posted June 29, 2011 Report Share Posted June 29, 2011 I have an entire manta bot available if interested?http://botguru.net/videos/manta/index.html I had someone interested in it and they have failed to get back to me. So im thinking about selling the source. Quote Link to post Share on other sites
LoWrIdErTJ - BotGuru 904 Posted June 29, 2011 Report Share Posted June 29, 2011 here is what your trying to do with the search. will gather all search resultsmanta.ubot Quote Link to post Share on other sites
mjmccoy 0 Posted June 29, 2011 Author Report Share Posted June 29, 2011 How much would you charge for the source? I would really like to learn ubot but also need to product this scrapper quick we are trying to build a database to cross reference with another database? M Quote Link to post Share on other sites
LoWrIdErTJ - BotGuru 904 Posted June 29, 2011 Report Share Posted June 29, 2011 sending you a PM now.. Quote Link to post Share on other sites
laurynherbert 0 Posted June 23, 2012 Report Share Posted June 23, 2012 here is what your trying to do with the search. will gather all search results I tried to download the file.. I'm having the same issue to (I'm new, just started today). Can you send me that file as well? It would really help me out. Thanks. Quote Link to post Share on other sites
LoWrIdErTJ - BotGuru 904 Posted June 23, 2012 Report Share Posted June 23, 2012 File is outdated, and from June of 2011 and no longer works on manta, as well was built in v3.5 of ubot. What is it your needing help with? Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.