Lemme Crash At Your Place. (Memory Resetting?)

[Biks 9]

I've had this issue since day one. I've probably created over 50 UBots so far, and it's rare that a bot won't crash after running overnight. I'm not even making complicated bots. It seems something is "piling up" in the background when they are running. Switching user agents every time it loops has helped a lot, but eventually it crashes.

 

Here's a specific example: I want to scrape Twitter users. Right around the 7,000 mark the bot locks up every time. (while switching user agents) Twitter only offers you a a single list of users (following or followed) that are only displayed on one screen. You have to use this to scroll down and load more:

 

run javascript("window.scrollTo(0,document.body.scrollHeight);")

 

I can't keep closing the page, or else I'll lose everything. The data isn't parsed out via single pages like some sites. It appears Ubot runs out of memory after a long scrape. What can I do in the background to coax it along? What are some other tricks to prevent crashing in general? Can I manually assign all my computers memory to Ubot?

[Brutal 164]

Biks, I have never been able to get one to run over night either. I have used the reset useragent trick along with clearing system memory with each loop - and just like you, never once had the stability to run over a few hours without crashing. - I think it's just the nature of the beast and ultimately something we have to live with. I hope that isn't the case, but I have been using ubot for years and that point has never changed.

[nuc63 27]

Every 1000 entries clear list and switch to a different file.( increment file name)

Scrape1 scrape 2 etc. And same with with importing.

It's extra coding but it's more stable.

[luis carlos 94]

For this specific case, I use the mobile version.

[HelloInsomnia 1103]

Try this: http://network.ubotstudio.com/forum/index.php/topic/18303-extremely-elegant-solution-to-rising-browserexe-memory/

 

I suppose it wont work in your Twitter example but for other bots it will.

[jason 101]

This has the best title of any post i've ever seen. A+

 

I had a bot that was crashing once - it was driving me crazy. It’d run for almost exactly 3-4 hours, it was scraping data, and it would either slow down a lot at some point or crash (usually it just slowed down). 

 

I just kept restarting it like you probably do - then, i realized that instead of dumping the data into a file and clearing every time a new page loaded (which was what i intended), it was just adding to this huge table inside UBot's memory. It was also saving to a file, but not clearing. UBot hated that, and it was poor programming practice. I fixed it in 10 minutes and that bot ran for 2 days until it scraped the entire site.

 

Not sure if you might be holding information in your lists/tables/variables and having it grow, but if you are, make sure to clear it regularly. 

 

As far as having this issue inside the browser because you're scrolling down and down and down - I imagine this problem would exist if you did it in any browser, such as Chrome or firefox, just because of what you're basically doing.

 

Not sure if there's an obvious solution for that.  

[cүвεя_נυηкιε 68]

Hi Biks,

 

Ive also created a shed load of bots and most of them will crash if left long enough, .. 

however, I have 1 bot that has been running for 9 months now, and by 9 months, I mean 9 months non stop (and no, i dont turn some of my PC's off ... ever lol), 

admittedly it doesnt do a hell of a lot of scraping but it is still running as i type,

 

The fix for me on this occasion was a simple blank load html node at the end of its particular process/run, ..

the moment the load html node is reached any browser memory is instantly cleared/reduced,

 

Im not saying this will fix your issue but thought i would mention it just in case you had not already.

 

 

Also probably worth mentioning to check out some tests with your User agents, Ive lost count the amount of customers that have reported dramatic improvements, 

and equally dramatic decreases in stability and performance  just from switching it.

 

 

 

well thats my 2 cents lol, 

hope you get it fixed

 

 

Cheers

CJ

[Biks 9]

OK, so I was able to scrape 50,000 + Twitter accounts by being in mobile mode (didn't know why I didn't think of that earlier) and saving and clearing the scraped list after every 1,000. (as mentioned above) Thanks.

 

 

The fix for me on this occasion was a simple blank load html node at the end of its particular process/run,

 

This is interesting. We're talking this: load html($nothing) Right? I'll have to give that a try.

 

It's 'nice' to know that everyone else's bots 'deflate' after a while. It just wasn't me being the world's crappiest programmer.

[cүвεя_נυηкιε 68]

OK, so I was able to scrape 50,000 + Twitter accounts by being in mobile mode (didn't know why I didn't think of that earlier) and saving and clearing the scraped list after every 1,000. (as mentioned above) Thanks.

 

 

This is interesting. We're talking this: load html($nothing) Right? I'll have to give that a try.

 

It's 'nice' to know that everyone else's bots 'deflate' after a while. It just wasn't me being the world's crappiest programmer.

 

 

Yep absolutely correct, .. in fact you dont even need to put the "$nothing" and can be left blank

although admittedly I did add a little "Please Wait ... ", you know just in case the user was watching lmao

 

lol an no dont think you are "the world's crappiest programmer", it really depends on the process being run, 

most bots that have only a few processes tend to run without any problems, its when you start looping shit that it becomes unstable ... not matter how clean your code is, 

 

for example my bot Easy-Crypt only processes a couple of tasks and has never crashed or hogged memory etc, 

but most of my other bots that loop a process until the cows come home will usually crash after a couple of hours unless i use a load html in the right place, 

and again it still is not guaranteed to 100% resolve the issue as there are too many variables that could cause a crash .. thread drop, browser crash, variable leakage, a site timing out when loading etc etc etc etc etc ... 

 

either way glad you are getting somewhere

 

 

 

 

CJ

[Biks 9]

I'm still having the same old issues with crashing even after trying load html($nothing). It's pretty consistent after so many iterations.

 

I guess the final solution is to restart ubot in the middle of a sequence. I've never done that.

 

Anyone have any example code of how this is done? Has someone covered this in another thread? I've done a search for "restarting", but can't find anything specific.

[cүвεя_נυηкιε 68]

I'm still having the same old issues with crashing even after trying load html($nothing). It's pretty consistent after so many iterations.

 

I guess the final solution is to restart ubot in the middle of a sequence. I've never done that.

 

Anyone have any example code of how this is done? Has someone covered this in another thread? I've done a search for "restarting", but can't find anything specific.

 

 

I have a sort of "semi" solution that im currently in the process of coding for 2 of my bots that crash no matter what extra implementations i add, ... 

maybe it will work for you, .. Im not sure as im not 100% sure what your bots process is, ...

 

I have a bot that loops continuously using multiple threads and most users of it say it crashes without fail around the 3-5 hour mark (on average)

Ive added/modified/optimised/ it so much that it really has no where else to go and is as "efficiently" coded as its gonna get, 

so after a couple of head scratching days of re-reading through the code I decided to take a look at scheduling,

and no not UBot's scheduling simply because I "think" the bot needs to remain open to be automated which kind of defeats the issue lol, 

 

So I tested auto opening via Windows task scheduler and as mentioned in a ton of pages here the "/auto" trick doesnt work any more on any UBot 5 version, 

so if you dont mind coding a little "If Schedule = True - Then Continue this process etc .." within an "On-Load" node then it can be the miracle your looking for, 

its perfect because you can set it to fire off at set intervals, but even better you can force a currently running instance to close before starting the new,

It could get a bit complicated telling your bot to pick up where it left off but if you save stuff to a file as you go along then implementing it shouldnt be too much of a headache.

 

 

So in my case I have built a scheduling system that gives my user the option to have the bot run between certain times of day, for specific length of time,

also the length of time the bot will run before closing the current running instance and then re-starting the next, ..

eg.

Start every day at 9am - Finnish at 9pm

Open and run for 1 hour internals, close and restart

 

So far in testing it runs as set, and on the off chance it crashes within the first hour the scheduler doesnt  have any problems forcing it to close,

also in my case it doesnt matter where it starts so nothing to remember when restarted.

 

 

 

There is a plugin here somewhere that will restart the bot for you if it crashes which would be alot simpler than my method, 

but I couldnt get it to work because when testing in a bot and it crashes it doesnt close for me unless i close it via task manager. 

 

You could also make it restart without scheduling and fire off a .bat file at the end of a timer or set amount of loops then use a "Close Bot" plugin (a few here), 

then within the bat file have your bot called, this way you would still need to "save your position" for the next restart, .. 

drawback to this option is if your bot crashes then it will never fire off the bat file or close :/

 

 

 

anywho, just a few suggestions i thought i would air

you never know something might click when you read it 

 

 

 

 

Cheers

CJ

[Biks 9]

 

There is a plugin here somewhere that will restart the bot for you if it crashes which would be alot simpler than my method,

 

OK, I've yet to install one plug in. I'm rummaging through the store, I don't see anything specific to restarting. Is it part of a larger plug in?

[HelloInsomnia 1103]

OK, I've yet to install one plug in. I'm rummaging through the store, I don't see anything specific to restarting. Is it part of a larger plug in?

 

I'm not sure which plugin it is but it is probably listed here: http://network.ubotstudio.com/forum/index.php/topic/13435-every-plugin-listed-here-selling-buying-look-no-further/

[cүвεя_נυηкιε 68]

OK, I've yet to install one plug in. I'm rummaging through the store, I don't see anything specific to restarting. Is it part of a larger plug in?

 

 

here it is and a few others, all available on the page HelloInsomnia posted

 

 

 

Name of plugin: Close bot

Link to the plugin thread: http://www.ubotstudi...se-bot-command/

 

 

 

Name of plugin: Advanced ubot (set 1)

Link to the plugin thread: http://www.ubotstudi...rograms-easier/

 

• exit this app: Force Quit Applications.
• restart this app: restart your program.

 

 

 

Name of plugin: Advanced Ubot 2

Link to the sales thread: http://www.ubotstudi...3-day-discount/

 

• $app crash check : last start app crash or not (Beta).
• app crash restart : auto restart when app crash (Beta).
• app crash test : test crash even (Beta).
• on closing : run ubot command on bot closing.

 

 

 

 

 

CJ

[Biks 9]

OK, I'm guessing that this is what I want. Pash's Advanced Ubot 2

 

http://imgur.com/x5mwwj7.png

 

Can anyone confirm that the restart aspect works? It's says beta. I thought somebody said it gets stuck and you have to kill the background process manually.