scrolling to bottom of page that keeps loading....

[wouldjaball 5]

Hi all,

I am trying to get the bot to scroll to the bottom of this page www.facebook.com/friends/requests   when I put in a first  or last name.   I'm currently using the javascript code below to do it:

 

loop(100) {
    run javascript("
window.scrollTo(0,document.body.scrollHeight)")
    wait(1)

 

 

as you can see I'm currently using a loop, but I'm trying to find a way to  keep scrolling until there aren't any more entries without limiting to a loop, as sometimes there are still more entires, and sometimes there aren't 100, and the bot stalls until all the loops are completes.. Is this possible?

Any help would be appreciated!

 

Thanks!

[LoWrIdErTJ - BotGuru 904]

loop while($not($exist("element here that shows at the end of the page"))){
run javascript( window.scrollTo(0,document.body.scrollHeight); )
}

 

can monitor the list total if its the same as the previous loop then its likely reached the end, but more difficult to setup to monitor it.

 

[wouldjaball 5]

Thanks bot gurue, but the problem is there is nothing at the end of the pages that signifies the end, it just stops scrolling... Any ideas?

[LoWrIdErTJ - BotGuru 904]

your scraping something to a list?

add list to list before the loop while
and scrape page results

set var = 0

loop while ($compaison(#var != $list total))
run javascript to bottom of page
set variable (list total)
add list to list, scrap items


it will auto exit after var value = the list total of the list.

because var is set to list total before it scrapes they should be different if it finds more info to scrape.

else on next loop they will end up equal when there is nothing changed and exit the loop while.

[blumi40 222]

run javascript( window.scrollTo(0,document.body.scrollHeight))

[wouldjaball 5]

botguru,

you are correct, but not sure I completely understand... I don't have a loop while...

Edited January 27, 2014 by wouldjaball

[wouldjaball 5]

Botguru,

I am having trouble, not sure what I am missing.  I have the code set like this however, it doesn't seem to loop.  If I switch the last two lines of code, everything works OK, until the bot gets stuck at the end and can't go any further.  Can you see what I'm doing wrong?

 

set(#Loop, 0, "Global")
loop while($comparison(#Loop, "=", $list total(%LastNames))) {
    run javascript("
window.scrollTo(0,document.body.scrollHeight)")
    set(#Loop, $list total(%LastNames), "Global")
    add list to list(%LastNames, $scrape attribute(<href=w"https://www.facebook.com/*?">, "href"), "Delete", "Global")
}

[Bot-Factory 602]

He's trying to scrape stuff from facebook.

And facebook has some dynamic stuff on their website.

 

They show new entries as soon as you scroll to the bottom of the page. 

When you reach the bottom, it will load and show new entries. 

 

So you don't know the total number of elements on that page at the beginning. 

 

I haven't looked at the facebook site myself, but i think you have to exit the loop depending on the scrape results. 

If you find new entries with your scrape command, do one more loop. If there are no more entries, exit the loop.

 

Dan

[LoWrIdErTJ - BotGuru 904]

[[WAIT FOR PAGE TO LOAD HERE MAYBE 2-5 seconds]]
add list to list(%LastNames, $scrape attribute(<href=w"https://www.facebook.com/*?">, "href"), "Delete", "Global")
set(#Loop, 0, "Global")

loop while($comparison(#Loop, "=", $list total(%LastNames))) {
    run javascript("
window.scrollTo(0,document.body.scrollHeight)")
    [[WAIT FOR PAGE TO LOAD HERE MAYBE 3-5 seconds]]
set(#Loop, $list total(%LastNames), "Global")
    add list to list(%LastNames, $scrape attribute(<href=w"https://www.facebook.com/*?">, "href"), "Delete", "Global")
}




This is how your code should look

[keyuramins 1]

Thanks guys for this useful post. Got my work accomplished due to your help.

[Brutal 164]

I'm having the same issue with a google page i'm trying to scrape - unfortunately none of these solutions work for me. Any other ideas?

[UBotDev 276]

I'm having the same issue with a google page i'm trying to scrape - unfortunately none of these solutions work for me. Any other ideas?

What's the problem on your side?

 

The approach that TJ published above (loop while ($compaison(#var != $list total))) should work for most of the cases. The bot should stay in the loop for the time when discovering new items, but as soon list total stays the same for 2 cycles, the loop will quit and you shouldn't have any new items displayed after that.