Josh 37 Posted February 24, 2013 Report Share Posted February 24, 2013 So I came up with a small work around when scraping gigantic lists to reduce the memory used in my bots and keeps them from crashing. Before this fix the bots would get to about 40k scraped urls and then freeze up and the only way to even shut them down would be to end the process in the task manager. The work around is basically this... once the list reaches 10K it saves the list to a text file. The end result being a bunch of numbered text files like search-1, search-2, search-3 etc. Previously my stat monitor was tracking the total number of urls scraped by exactly that.... setting the stat monitor variable to the total number of items contained in that list... Now that I am saving that list to a text file and then clearing the list the stat monitor is only showing the current number of urls in the list rather than the total number of urls that have been scraped. Can anyone think of a good way to keep track of the total number of urls scraped when the list is being cleared every so often? Quote Link to post Share on other sites
AutomationNinja 194 Posted February 24, 2013 Report Share Posted February 24, 2013 Have another list with the all of the scraped lists Quote Link to post Share on other sites
Josh 37 Posted February 24, 2013 Author Report Share Posted February 24, 2013 Have another list with the all of the scraped lists That sort of defeats the purpose of saving and clearing the first list. The larger the list gets the more it tends to cause problems. Which is why I save and clear the list every 10K. Quote Link to post Share on other sites
Lucius 7 Posted February 24, 2013 Report Share Posted February 24, 2013 save all your data to a folder... have a loop command for ubot load each txt file into a temp list... then count the total of each load... save the total number.. and do a simple addition of all the totals in the end? Quote Link to post Share on other sites
jomark3 25 Posted February 25, 2013 Report Share Posted February 25, 2013 In the first part of your bot, Set a variable #numberoflists = 0Just before saving and deleting the list, increment #numberoflistsWhen the bot ends, display #numberoflists or list total = #numberoflists times 10,000 Quote Link to post Share on other sites
Josh 37 Posted February 25, 2013 Author Report Share Posted February 25, 2013 In the first part of your bot, Set a variable #numberoflists = 0Just before saving and deleting the list, increment #numberoflistsWhen the bot ends, display #numberoflists or list total = #numberoflists times 10,000 Yeah I thought of that, but that doesn't allow monitoring of the number in real time. Which is what I'm trying to do. Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.