Jaro 6 Posted September 20, 2017 Report Share Posted September 20, 2017 Hi there, I have found a bug in the Ubot software where the browser function $document text doesn't refresh after a page scroll. I'm trying to scroll down and scrape a page but I found out that it always scrapes only the first 22 results even though I see the page being scrolled down and see all the other results. Ubot, in the loop scrapes just the first 22 results because it scrapes from $document text which doesn't refresh itself after the scroll. When I manually check the source code, it shows the unrefreshed content - the same one as $document text and when I check the view generated source, it correctly shows the next loaded page with new data. So the question is: how can I scrape from the generated source code which is being refreshed as opposed to the classic source code ($document text)? Thanks so much in advance for any help! Quote Link to post Share on other sites
pash 504 Posted September 21, 2017 Report Share Posted September 21, 2017 try post your code. Quote Link to post Share on other sites
HelloInsomnia 1103 Posted September 22, 2017 Report Share Posted September 22, 2017 Usually you can look in Fiddler or in the Network tab of Chrome Dev Tools while scrolling down the page to see how the content is being loaded and then scrape it from there. Quote Link to post Share on other sites
Jaro 6 Posted September 22, 2017 Author Report Share Posted September 22, 2017 Yes, I have actually started doing it differently and now I just don't use the $document text function but scrape the current source code and then scrape the necessary text from there. 1 Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.