mrwalt 34 Posted May 5, 2014 Report Share Posted May 5, 2014 Hello Ubotters, I was curious does anyone know of a way if you have a couple of large list of lets say 10,000 items each that you are extracting from a page besides the Advanced Filefree plugin that can handle that amount of data? I know that the Advanced File free plugin it has a clipboard function but that only holds data for one list at a time. What if you have multiple list that you are trying to pull data into from a page how would you mange this? I know that there has to be a way and I would greatly appreciate any feedback on how you could work around this since after you start gathering a couple of thousand items to the list Ubot becomes very slow and unresponsive and at a certain point will just stop or delete the data from the list. Thank you, mrwalt Quote Link to post Share on other sites
Bot-Factory 602 Posted May 5, 2014 Report Share Posted May 5, 2014 Hi Mrwalt, list work fine in Ubot. The problem normally only occurs in the UI and when you look at the debugger. If the list is open in the debugger, Ubot get's very slow and squishy quickly. In a compiled bot I had now issues to work with lists up to 50.000 entries. If recently built a scraper that extracted 2 Million Urls. I was using sqlite for that. So I scraped 10000 Urls, saved them to the database, cleared the list and repeated that multiple times. That was working fine. Dan Quote Link to post Share on other sites
Code Docta (Nick C.) 638 Posted May 5, 2014 Report Share Posted May 5, 2014 or you can use this free plugin http://www.ubotstudio.com/forum/index.php?/topic/16308-free-plugin-large-data/ 1 Quote Link to post Share on other sites
mrwalt 34 Posted May 6, 2014 Author Report Share Posted May 6, 2014 Thank you both for your feedback and I will try out the free plugin to see if it helps, as right now when I scrape data andadd it to a list that gets too big it is very problematic for UBot for some reason. mrwalt Quote Link to post Share on other sites
blumi40 222 Posted May 6, 2014 Report Share Posted May 6, 2014 mrwalt 10k is not that big. but yes ubot needs a while to load it.your texteditor needs a while too if u load a big textfile thereso give ubot a bit to doing that Quote Link to post Share on other sites
Code Docta (Nick C.) 638 Posted May 6, 2014 Report Share Posted May 6, 2014 also append to file, just fo str8 to a file with it Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.