Anyway to load/scrape from unlimited loading web 2.0 type pages?

[zonfar 1]

A lot of websites are now gearing towards the style of scrolling down to load extra content instead of just having a next page button which is hell for automation. I wasn't sure if there was some sneaky way of loading all this content to scrape it all. I've used javascript to scroll down, wait, scrape, repeat. However after doing this "x" amount of times too much is loaded on the page and eventually ubot freezes.

[UBotDev 276]

I don't think there is another way around that.

 

I know Twitter displays user feeds like that, and in that case I think you could get around by scraping regularly, so you don't have to go as deep as with a single scrape.

[rocket976 62]

just loop while "more" or whatever it is exists, then throw a click in there or can also use java or jquery

 

run javascript("for (i = 0; i < 100; i++) \{
    document.write(\"\" + i + \"<br />\");
    window.scroll(0,document.body.offsetHeight);
\}

")
 

Could have swore this worked for me before but now its showing the numbers.. maybe a java pro can chime in here..