General Lee 12 Posted August 27, 2012 Report Share Posted August 27, 2012 Yes, the ol' confederate general has many questions and I do appreciate the continued support. I'm trying to scrape some domains on a site that has them listed in single colums of 30 spread out through 20 pages. Formated like page (1) (2) (3) etc etc but there is not a next page button so how do I scrape these 20 pages in a loop? I'd hate to have to code each page..ugh Quote Link to post Share on other sites
k1lv9h 76 Posted August 27, 2012 Report Share Posted August 27, 2012 Hi, When you go from page to page does the URL have information as to what page is being displayed? If it does then maybe you could:loopnavigate to url?displaywhateverpage=xscrape information Kevin Quote Link to post Share on other sites
General Lee 12 Posted August 27, 2012 Author Report Share Posted August 27, 2012 Kevin, Thanks for the reply, I'll have to double check the url. I was thinking there maybe a way to increment the count up in some fashion but I'm not quite capable. Quote Link to post Share on other sites
blumi40 222 Posted August 27, 2012 Report Share Posted August 27, 2012 Maybe my Example with Google brings ya a light navigate("http://www.google.com", "Wait") type text(<name="q">, "ubot", "Standard") click(<name="btnK">, "Left Click", "No") wait for browser event("Page Loaded", 5) add list to list(%nexturl, $scrape attribute(<class="fl">, "fullhref"), "Delete", "Global") Quote Link to post Share on other sites
LoWrIdErTJ - BotGuru 904 Posted August 27, 2012 Report Share Posted August 27, 2012 Showing a couple different ways to do it. set(#var, 1, "Global") comment("scrape the number of pages or a set ui element or number of pages") loop(#pages) { navigate("http://www.ubotstudio.com/resources?page={#var}", "Wait") increment(#var) } divider divider set(#var, 1, "Global") comment("scrape the number of pages or a set ui element or number of pages") loop(#pages) { click(<innertext="( {#var} )">, "Left Click", "No") increment(#var) } Quote Link to post Share on other sites
blumi40 222 Posted August 27, 2012 Report Share Posted August 27, 2012 poser :) :) Quote Link to post Share on other sites
LoWrIdErTJ - BotGuru 904 Posted August 27, 2012 Report Share Posted August 27, 2012 LMAO Quote Link to post Share on other sites
General Lee 12 Posted August 28, 2012 Author Report Share Posted August 28, 2012 Thanks guys! I'm working my through it with the codes you provided.. Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.