How To Deal With Very Large List?

[mrwalt 34]

Hello Ubotters,

 

I was curious does anyone know of a way if you have a couple of large list of lets say 10,000 items each that you are extracting from a page besides the Advanced File

free plugin that can handle that amount of data?  I know that the Advanced File free plugin it has a clipboard function but that only holds data for one list at a time.  

What if you have multiple list that you are trying to pull data into from a page how would you mange this?  

 

I know that there has to be a way and I would greatly appreciate any feedback on how you could work around this since after you start gathering a couple 

of thousand items to the list Ubot becomes very slow and unresponsive and at a certain point will just stop or delete the data from the list.

 

 

Thank you,

 

mrwalt

 

 

 

[Bot-Factory 602]

Hi Mrwalt,

 

list work fine in Ubot. The problem normally only occurs in the UI and when you look at the debugger. 

If the list is open in the debugger, Ubot get's very slow and squishy quickly.

 

In a compiled bot I had now issues to work with lists up to 50.000 entries.

 

If recently built a scraper that extracted 2 Million Urls. I was using sqlite for that. 

So I scraped 10000 Urls, saved them to the database, cleared the list and repeated that multiple times.

 

That was working fine. 

 

Dan

[Code Docta (Nick C.) 638]

or you can use this free plugin

 

http://www.ubotstudio.com/forum/index.php?/topic/16308-free-plugin-large-data/

[mrwalt 34]

Thank you both for your feedback and I will try out the free plugin to see if it helps, as right now when I scrape data and

add it to a list that gets too big it is very problematic for UBot for some reason.

 

 

mrwalt

[blumi40 222]

mrwalt 10k is not that big. but yes ubot needs a while to load it.
your texteditor needs a while too if u load a big textfile there

so give ubot a bit to doing that

[Code Docta (Nick C.) 638]

also append to file, just fo str8 to a file with it