Scrolling webpage before scrape => Twitter changes

[Jon12345 0]

When I go to Twitter, I see 20 odd tweets ready to scrape. But if I scroll to the bottom of the page, it keeps adding more Tweets. Before, I am sure there was a button that you used to click for more tweets, right?

 

So what can you do now? If I want to scrape I sizeable page of tweets, I figure I will need to load the page and then scroll that page a few times, then do the scrape.

 

How do I get ubot to scroll so more tweets are added?

 

Thanks,

 

Jon

[turbolapp 5]

run javascript

location.reload()

 

Assuming you're searching through popular results, like trends or something, you need only to refresh the page for new results.

[JohnB 255]

I just logged in and it looks like it just keeps loading and loading...you may just have to put in a significant delay and wait for them (i.e. 60 seconds or more)

 

John

[LoWrIdErTJ - BotGuru 904]

run javascript

window.scrollTo(0,document.body.scrollHeight);

delay or wait finish

run it again

window.scrollTo(0,document.body.scrollHeight);

delay or wait finish

 

repeat process as many times as you want till enough are showing.

 

Have fun.

 

TJ

[amul 1]

run javascript

window.scrollTo(0,document.body.scrollHeight);

delay or wait finish

run it again

window.scrollTo(0,document.body.scrollHeight);

delay or wait finish

 

repeat process as many times as you want till enough are showing.

 

Have fun.

 

TJ

 

 

I was trying to figure out how to do get javascript to load this..

Now I've to do some regex for friend ids.. to message.

 

thanks TJ!!