stanf 43 Posted October 25, 2012 Report Share Posted October 25, 2012 i am a standard ubot userall i am doing is scraping a few sites for my (hopefully) soon to open ecom site, ( although this christmas season will probably be a wash).I try to keep my loops under 1000 reps on a laptop with xp and 2 gigs of memmory (still crashes 6 out of 10 trys).i have clearmem set to wipe the memmory every 3 minutes.i run the javascripts clear my tracks after every rep ( so you know scraping aprox 15000 pages takes forever).what else can i do to make my time productive,while the dev crew works out the bugs in the next release? 1-shorten the loops?2- improve my code logic?3 anything you can think of? thanxstanfcheap-bag-scrape1.ubot Quote Link to post Share on other sites
Legend 181 Posted October 25, 2012 Report Share Posted October 25, 2012 - you can try using a more powerful computer or get a couple more gigs of ram and see if that helps... unfortunately 2 gigs is pretty lame nowadays... - you can make sure all your drivers are up to date - you can try slowing down your processes (add more/longer internal waits) - you can try a shorter interval for clearmem Quote Link to post Share on other sites
a2mateit 395 Posted October 25, 2012 Report Share Posted October 25, 2012 Unfortunately you are in the same boat as we all are in... You simply cannot loop thousands of times without crashing and errors. It could also be the way you are managing your data with your scrapes. Are you saving the list everytime after a scrape, then clearing that list, then adding that list back to your list before your next scrape? Or are you just letting the list build and build and build and then finally trying to save it at the end of the loop cycle? One thing that you can try is breaking down your loop cycles into a series of bots, and then shell out the start of one after and close the other.?. If someone can prove me wrong and show me a bot that can successfully make thousands of loops right now then please blow my mind. Quote Link to post Share on other sites
Pete 121 Posted October 25, 2012 Report Share Posted October 25, 2012 improve my code logic? Yeslook at the first loopthen navthen wait 3 secsthen a whileloopthen wait 3 secs window.location.reload (see http://www.w3schools..._loc_reload.asp i think you should be using location.reload(forceGet) (But i'm not good with java so i keep as away form it if i can)here there should be a wait command i thinkBelow this are more "window.location.reload" without any wait commandsAlso you should use the defines to split the code into more manageable chunks You simply cannot loop thousands of times without crashing and errorsWhy ? Quote Link to post Share on other sites
stanf 43 Posted October 25, 2012 Author Report Share Posted October 25, 2012 a2mateit ----i took your advice and split the number of bots from 3 to 6 (less pages to work before closing the browser and opening a new one)the list only grows to a max of 700 lines before being written to a file and a new list is begun zap----thanks for the reload (force get) tipa lot of the waits are because1) i did not know what the problem was2) if i bang that site too fast i get a lot more page load errors but some of that i can correct because now i have a better understanding of how to check for and handle page load errors. thanx for the feedback fellas stanf Quote Link to post Share on other sites
Pete 121 Posted October 25, 2012 Report Share Posted October 25, 2012 a lot of the waits are because1) i did not know what the problem was2) if i bang that site too fast i get a lot more page load errors but some of that i can correct because now i have a better understanding of how to check for and handle page load errors. Sorry my point was you had no 'page load waits' after the Refresh Page so the bot continual before the page had loaded Quote Link to post Share on other sites
stanf 43 Posted October 25, 2012 Author Report Share Posted October 25, 2012 Ok, I think I see your pointsome of my crashes could have been because the bot had no data to work.is that what your saying? Quote Link to post Share on other sites
a2mateit 395 Posted October 25, 2012 Report Share Posted October 25, 2012 Why ? You tell me... Show me a bot that can loop thousands of times without crashing. I dare ya Quote Link to post Share on other sites
Pete 121 Posted October 25, 2012 Report Share Posted October 25, 2012 I don't sell bots so how? Quote Link to post Share on other sites
a2mateit 395 Posted October 25, 2012 Report Share Posted October 25, 2012 I was just saying that the "Awesomium" issue causes many lockups and crashes and it is very difficult to loop thousands of times. Especially when managing data, or if you have loops inside of loops... Quote Link to post Share on other sites
Pete 121 Posted October 25, 2012 Report Share Posted October 25, 2012 Awesomium is a problem but can be overcome, good code logic stops the lockup, and if you vist 20/30 thousand sites you may drop one or two threads but thats it Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.