Scraping multi pages in a loop with NO next page button?

[General Lee 12]

Yes, the ol' confederate general has many questions and I do appreciate the continued support.

 

I'm trying to scrape some domains on a site that has them listed in single colums of 30 spread out through 20 pages. Formated like page (1) (2) (3) etc etc but there is not a next page button so how do I scrape these 20 pages in a loop? I'd hate to have to code each page..ugh

[k1lv9h 76]

Hi,

 

When you go from page to page does the URL have information as to what page is being displayed?

 

If it does then maybe you could:

loop

navigate to url?displaywhateverpage=x

scrape information

 

Kevin

[General Lee 12]

Kevin,

 

Thanks for the reply, I'll have to double check the url. I was thinking there maybe a way to increment the count up in some fashion but I'm not quite capable.

[blumi40 222]

Maybe my Example with Google brings ya a light

 

navigate("http://www.google.com", "Wait")
type text(<name="q">, "ubot", "Standard")
click(<name="btnK">, "Left Click", "No")
wait for browser event("Page Loaded", 5)
add list to list(%nexturl, $scrape attribute(<class="fl">, "fullhref"), "Delete", "Global")

[LoWrIdErTJ - BotGuru 904]

Showing a couple different ways to do it.

 

set(#var, 1, "Global")
comment("scrape the number of pages or a set ui element or number of pages")
loop(#pages) {
   navigate("http://www.ubotstudio.com/resources?page={#var}", "Wait")
   increment(#var)
}
divider
divider
set(#var, 1, "Global")
comment("scrape the number of pages or a set ui element or number of pages")
loop(#pages) {
   click(<innertext="( {#var} )">, "Left Click", "No")
   increment(#var)
}

 

[blumi40 222]

poser :) :)

[LoWrIdErTJ - BotGuru 904]

LMAO

 

[General Lee 12]

Thanks guys! I'm working my through it with the codes you provided..