How To Stop Scraping Scrolling Down Pages

[webpro 31]

loop(20) {
    run javascript("window.scrollTo(0, document.body.scrollHeight);")
    wait(3)
}

This is what i am using right now, to scroll down pages like Pinterest. You know when stuff is always added to the page as you scroll down ?

The thing is, i want the bot to stop when there's nothing else to add but the problem is, there's nothing to tell the bot that it has reached the bottom.

Well nothing i could see that would make it stop.

Like use a WAIT FOR ELEMENT command or something.

 

How would you do this ?

Is there a "if there's nothing new to add to a list stop". Command or trick ?

 

Heck as i was typing this i just figured that maybe that would do the trick (unless you have a better idea?)

 

You see, i'm scraping profiles with a regex i came up with and it adds them to a list alright.

So i could tell the bot that, if no new addition after 3 scroll down, then stop (or move on to do whatever else)

Mind sharing a code idea ?

 

Thanks guys

[deliter 203]

**edit,made a mistake,adjusted code**

this works ok on pinterest

 

 

set(#atBottom,0,"Global")
loop while($comparison(#atBottom,"< Less than",1)) {
    set(#before,$document text,"Global")
    run javascript("window.scrollTo(0, document.body.scrollHeight);")
    wait(3)
    wait for browser event("DOM Ready","")
    wait for browser event("Everything Loaded","")
    set(#after,$document text,"Global")
    if($comparison(#before,"=",#after)) {
        then {
            set(#atBottom,1,"Global")
        }
        else {
        }
    }
}

[webpro 31]

THANKS A LOT MATE!

By the way, is this working for you

 wait for browser event("DOM Ready","")
    wait for browser event("Everything Loaded","")

ain't talking about your example here but in general?

Looks like everytime i want to use one of these (or both) it never waits?

It moves on to the other node no matter what ?

 

 

By the way if anyone else has a trick, shoot please!

Thanks

[deliter 203]

they are a bit hit or miss,I like to add this define to bots I make,but their are better ways of waiting,like wait for element

 

define WaitForAll {
    wait for browser event("Everything Loaded","")
    wait for browser event("Page Loaded","")
    wait for browser event("DOM Ready","")
}
WaitForAll()

[webpro 31]

Ohhhhhhhhhhhhhhhhh so clever!

Why i love this board!

So simple and yet so effective!

 

I'm better with a guitar/keyboard and a mic i guess.

Oh well, i'll get the hang of it in a decade from here or so LOL!

Thanks deliter