Wait For Element - Waiting For Page To Load?

[InfernusDoleo 0]

Quick question regarding wait for element. I'm doing some scripting on a webpage which, after the page loads, some javascript fires off that loads some additional information. Basically - the page loads, then after the page (and the elements I need are available) is loaded, some ajax calls are made that, using your IP address, loads some extra data to the page.

 

I do not need to wait for all the extra ajax calls to run for me to act on the page, yet, even though I am using 'wait for element' to wait for the page to load the button I need, the script is not continuing until all of the extra javascript/ajax elements are loaded. Does wait for element not proceed until the page is completely loaded? I'm not doing a navigate which requires the page to load - I'm clicking a button, which then loads a new page. I can proceed before the rest is loaded, as long as the next button is there. However, wait for element seems to sit and wait for all ajax calls to complete before continuing.

 

Is that normal? And if so, can someone suggest a workaround? I'm running my script through a TOR socks proxy, which slows the whole process down considerably. The loop of my script, since it's running through tor, is delayed by about 3-4 seconds per loop because of the waiting to load. I've tried to find where the calls are being made to block the javascript from running, but the page is extremely complex and it would take a long while to find the code. 

 

Is there another option? If I don't wait, since it's through tor, the script runs before the page is fully loaded. But I dont want to wait for it to load completely before continuing. Suggestions?

 

The page in question is lowes.com - I'm scraping product data. After the page loads, it then uses your IP or zipcode to show you your local store. If you load it through a slow connection (TOR), you can see at the top, 'Welcome to Lowes, Find a Store', then a few seconds later as the page continues to load, it changes to "Open until 10PM! <your town> Lowe's" I don't need to wait for that to proceed, yet, wait for element is not continuing until that is finished loading.

[pash 504]

default i use

navigate("http://www.ubotstudio.com/resources","Wait")
define Wait1 {
    wait for browser event("Everything Loaded","")
    wait(1)
    loop(60) {
        if($not($exists(<class="btn btn-primary">))) {
            then {
                wait(1)
            }
        }
    }
}
define Wait2 {
    wait for browser event("Everything Loaded","")
    wait for element(<class="btn btn-primary">,"","Appear")
}

[InfernusDoleo 0]

Yeah I guess I'll have to go that route. It does appear that wait for element is waiting for the entire page to load before checking. Making my own check loop will avoid that. Thanks.

[GhostBot 4]

I am not sure if I am having a similar issue. I am working towards scraping all 'javascript postcards' on twitter in a search. What I am striving to achieve is scraping everything on that search. I have used 'wait for browser event>Everything loaded' but I still only attain a few items on page.

On first attempt, I get 18 items but when I scroll down and page loads more data, I get 78 items when re-running the node. Is there a way to -scroll down til end - then scrape? 

Thanks! 

[GhostBot 4]

Ahhhh I think I found an answer on here. 

http://network.ubotstudio.com/forum/index.php/topic/19310-scraping-usernames-using-keywords-on-twitter/?hl=scroll+load&do=findComment&comment=120637

window.scrollTo(0,document.body.scrollHeight);

Let me try and if not return, this was the solution. 

Thanks fellow botters.