Script collects all the data only after several runs

[semenoffalex 0]

Hi!

 

I've written a simple script to get results from http://people.yandex.ru, but if i run it for the firs time after launching UBot Studio it scrapes only urls from first page. On the second run it gets data from 1st and 2nd pages and so on. Once it reaches the neccessary amount of pages/runs it starts working correctly and scrapes all the links in one run.

 

The workflow of the script is really simple:

navigate -> type text -> click (search button) -> wait for browser event (page loaded) -> clear list (%urls) -> loop while (exists -> wait for browser event (page loaded) -> add list to list -> click (next page of results) -> wait for browser event (page loaded)) -> wait for browser event (page loaded) -> add list to list  -> save to file

 

 

Do you have any idea, what can be wrong with that?

 

 

[Frank 177]

ok, the first issue is the way you are waiting for a page to load, 'wait for browser event (page loaded)'. This isn't 100% reliable as page elements can continue loading even after the page is technically loaded. You gotta love javascript! What you want to do is use the command 'wait for element' and select an element on the page that is below the info you want to scrape. This will ensure that the info is there when you scrape it. If you want even more security, add another command after the 'wait for element'. Just a 'wait' and wait .5 seconds.

 

Give it a try. And just don't bother with the wait for browser event commands anymore - at this time they are not reliable because of dynamic elements that continue to load after the dom is loaded.

 

Frank

[semenoffalex 0]

Thanks, Frank!

 

That works perfectly.