Help with browser crashing I think

[nicksavage 0]

Okay my bot is supposed to load a list of members, its a pop up and shows 11 members at a time, then you click load more and it shows another 11, well there are over 10,000 members and when it gets to about 900 ubot starts lagging bad, can anyone help me with this.

[HelloInsomnia 1103]

Can you post some code? That always helps.

[98quinnk 0]

Can you post some code? That always helps.

 

This is the code, im working along side of nick.

 

define Scrape Users {

    click(<class="user-list-link subs ud-popup">, "Left Click", "No")

    set(#start, $true, "Global")

    loop while($exists($element offset(<class="btn">, 1))) {

        then {

            click($element offset(<class="btn">, 1), "Left Click", "No")

            wait for browser event("Everything Loaded", "")

            wait(5)

            add list to list(%urls, $scrape attribute(<href=w"https://www.*.com/message/?action=send-new&userId=*">, "href"), "Delete", "Global")

            wait(20)

            else {

            }

        }

    }

}

[HelloInsomnia 1103]

If the bot just sits on a site and constantly collects data my guess would be either: the list gets to be too large - in which case you can save it off when it reaches a certain number (like a few thousand). Or, the browser.exe can also get too large so use in new browser to scrape and then after so many loops try close page (which will close that browser.exe instance) and then go back and scrape more.

[98quinnk 0]

Yes its the browser.exe getting too large, but if we close the browser and open it and list starts back at 12 and then it has to load more to scrape more again.

[HelloInsomnia 1103]

Yes its the browser.exe getting too large, but if we close the browser and open it and list starts back at 12 and then it has to load more to scrape more again.

 

What do you mean it starts back at 12? Sorry I am not quite sure what site this works on so I'm not quite sure whats going on.

[98quinnk 0]

On the site we are scraping members to message.

 

When we scrape members its a javascript popup on there website shows 12 users, and a loadmore button at the bottom.

we scrape the 12 the program presses load more and it shows the next 12 to be scrapped.

 

Gets to about 800 scrapped and then freezes up as it sky rockets the memory usage of browser.exe.

 

But if u close that window and reopen it , it shows the first 12 so all is lost.

[HelloInsomnia 1103]

On the site we are scraping members to message.

 

When we scrape members its a javascript popup on there website shows 12 users, and a loadmore button at the bottom.

we scrape the 12 the program presses load more and it shows the next 12 to be scrapped.

 

Gets to about 800 scrapped and then freezes up as it sky rockets the memory usage of browser.exe.

 

But if u close that window and reopen it , it shows the first 12 so all is lost.

 

Okay, I can see how that would be a problem. I can't think of anything off the top of my head since this seems to be an issue with Ubot (hopefully they fix it in Ubot 5)

[nicksavage 0]

I know its probably the browser, but even if I stopped it, then reloaded the browser, it would still have to load the first ones I saved to get to the others.

[nicksavage 0]

Does anyone know java, is there a way to get this popup to load in its own window at once with all members.

[HelloInsomnia 1103]

What site is it for? Without seeing the site its hard to say.

[UBotDev 276]

I noticed that this happens when browser.exe "in memory" size reaches a bit more than 1GB.

 

However, are "members" that you already scraped still displayed on the site/contained in its HTML?

 

If so you should remove them with "change attribute" command and that way reduce the HTML (size) that needs to be rendered by the UBot browser. This should reduce browser.exe RAM usage and prevent browser from crashing.

[nicksavage 0]

Its a java popup, there is no way to remove them. The browser is crashing due to the high load. and its not 1 gb its 8 gbs of memory being used at 900 loaded. I was hoping someone would know java and be able to tell me how to get around the browser problem, its not ubot problem but browsers.

 

I noticed that this happens when browser.exe "in memory" size reaches a bit more than 1GB.

 

However, are "members" that you already scraped still displayed on the site?

 

If so you should remove them with "change attribute" command and that way reduce the HTML (size) that needs to be rendered by the UBot browser. This should reduce browser.exe RAM usage and prevent browser from crashing.

[Code Docta (Nick C.) 638]

Try this thread

 

http://www.ubotstudio.com/forum/index.php?/topic/12812-example-popups-in-new-window-source/

[98quinnk 0]

I noticed that this happens when browser.exe "in memory" size reaches a bit more than 1GB.

 

However, are "members" that you already scraped still displayed on the site/contained in its HTML?

 

If so you should remove them with "change attribute" command and that way reduce the HTML (size) that needs to be rendered by the UBot browser. This should reduce browser.exe RAM usage and prevent browser from crashing.

 

 

They are not contained with in the html, its a java pop windows. so removing the members after scrape wont help.

[98quinnk 0]

How could I configure it,

 

So that it opens a new browsers scrapes 12 loads more. closes

Then goes back in?

 

That might keep the browser.exe down in memory usage.

 

Thanks

 

Kevin

[brusacco 20]

In each iteration, just append the list to a file, no need to keep it in memory till the hole scrap is terminated.

 

EDIT:

 

Well, each iteration you have to clear the list, so it wont get bigger. just append the results to the temp file.