kev123 132 Posted July 25, 2014 Author Report Share Posted July 25, 2014 Feature Request: 1.)Support for local ubot lists with the following command:plugin command("Bigtable.dll", "ubot list to large list", "scrapedata", %scrapedata) If I call that right after I store stuff in a local ubot list:add list to list(%scrapedata, $list from text($find regular expression($document text, #scraperegex), $new line), "Delete", "Local") It doesn't work 2.)Large list remove empty. Similar to large table remove empty Dan Feature Request: 1.)Support for local ubot lists with the following command:plugin command("Bigtable.dll", "ubot list to large list", "scrapedata", %scrapedata) If I call that right after I store stuff in a local ubot list:add list to list(%scrapedata, $list from text($find regular expression($document text, #scraperegex), $new line), "Delete", "Local") It doesn't work 2.)Large list remove empty. Similar to large table remove empty Dan1.hi dan there it's not possible to read a local list with the api but I can put in a work around by making the input a string or something. Quote Link to post Share on other sites
rocket976 62 Posted July 25, 2014 Report Share Posted July 25, 2014 How can I download the update Kev? Im already on your list , it wont let me subscribe again... Quote Link to post Share on other sites
Bot-Factory 602 Posted July 25, 2014 Report Share Posted July 25, 2014 How can I download the update Kev? Im already on your list , it wont let me subscribe again...You should have received an email. Dan Quote Link to post Share on other sites
Bot-Factory 602 Posted July 27, 2014 Report Share Posted July 27, 2014 Feature Request: Clear all large lists (Similar to Clear all Large Tables) ThanksDan Quote Link to post Share on other sites
kev123 132 Posted July 27, 2014 Author Report Share Posted July 27, 2014 added to the list thanks Feature Request: Clear all large lists (Similar to Clear all Large Tables) ThanksDan Quote Link to post Share on other sites
Bot-Factory 602 Posted July 27, 2014 Report Share Posted July 27, 2014 Wow. The performance of the plugin is really outstanding!!I have 76 million entries in a large list now. Plugin is using 7gb Memory now. And it's still running smoothly. Great job!Dan Quote Link to post Share on other sites
kev123 132 Posted July 27, 2014 Author Report Share Posted July 27, 2014 Wow. The performance of the plugin is really outstanding!!I have 76 million entries in a large list now. Plugin is using 7gb Memory now. And it's still running smoothly. Great job!Danwowtry sorting the list that's the ultimate test. I would save what your doing thou if you have 76 million records could get messy. 1 Quote Link to post Share on other sites
Bot-Factory 602 Posted July 27, 2014 Report Share Posted July 27, 2014 wowtry sorting the list that's the ultimate test. I would save what your doing thou if you have 76 million records could get messy.Removing duplicates works nice as well. I'm working on an universal scraping framework. using Http post to scrape data. And big table plugin to manage and save the data. I only enter the start url, the scrape regex and the nexturl regex. And hit run. Not sure how you handle that much data with great performance. When I think about the ubout table... Get's stuck after 20k entries already :-) Dan Quote Link to post Share on other sites
Code Docta (Nick C.) 638 Posted July 27, 2014 Report Share Posted July 27, 2014 Excellent work Kev123!!!!!!!!!!!!!!!! I really am thankful for this plugin and appreciate all that you have done with it and the other plugins you do... TC Quote Link to post Share on other sites
kev123 132 Posted July 27, 2014 Author Report Share Posted July 27, 2014 I'm working on an universal scraping framework. using Http post to scrape data. And big table plugin to manage and save the data. I only enter the start url, the scrape regex and the nexturl regex. And hit run. I have this done for search engines reduced my coding massively. Add a new search engine takes a few min's. Other benefit is you can encrypt the "data file" and save locally thus allowing silent updates. Quote Link to post Share on other sites
CTSchmidt 1 Posted August 7, 2014 Report Share Posted August 7, 2014 I have a question: how to i add a specific row from LargeTableA to LargeTableB? Quote Link to post Share on other sites
Bot-Factory 602 Posted August 8, 2014 Report Share Posted August 8, 2014 TIP: Currently the command:plugin command("Bigtable.dll", "ubot list to large list", "xx", %xx) Only supports global ubot lists. So if you want to convert a local list into a large list you can use: loop($list total(%tmp)) { plugin command("Bigtable.dll", "Add item to large list", "orderids", $next list item(%tmp)) } Kev already works on an update to address this. Quote Link to post Share on other sites
Bot-Factory 602 Posted August 8, 2014 Report Share Posted August 8, 2014 I have a question: how to i add a specific row from LargeTableA to LargeTableB? set(#loopcounter, 0, "Global")loop($plugin function("Bigtable.dll", "Large Table Total Rows", "TableA")) { plugin command("Bigtable.dll", "Set Large Table cell", "TableB", #loopcounter, 0, $plugin function("Bigtable.dll", "Large Table Cell", "TableA", #loopcounter, 0)) increment(#loopcounter)} CheersDan Quote Link to post Share on other sites
UBotBuddy 331 Posted August 8, 2014 Report Share Posted August 8, 2014 Hey Kev, I am a little late to the party but I just subscribed. I just blew out my UBot with a out of memory issue so I just happened to see this thread. I am looking forward to testing the Large List as well as the Large Table functions. Looks like it will save me a ton of coding work arounds. Later! Buddy Quote Link to post Share on other sites
kev123 132 Posted August 8, 2014 Author Report Share Posted August 8, 2014 Hey Kev, I am a little late to the party but I just subscribed. I just blew out my UBot with a out of memory issue so I just happened to see this thread. I am looking forward to testing the Large List as well as the Large Table functions. Looks like it will save me a ton of coding work arounds. Later! Buddygreat the reason it was created was to deal with two things list sizes and working in threads with these items globally. The next update (apart from a few additions ) is concentrated on using the tables and lists safely in threads at fast speeds this has come about from the smart thread plugin and the increased speed that threading is executed. Quote Link to post Share on other sites
CTSchmidt 1 Posted August 8, 2014 Report Share Posted August 8, 2014 (edited) Sorry i did a mistake, I meant: I have a LargeList with ID's (eg. 001, 002, 008) now i want to add a row from LargeTableA (with a matching id) to LargeTableB the result will be: LargeTableB: 001,data,data 002,data,data 008,data,dataThis was what I meant with "specific row" Thanks in advance CTSchmidt Edited August 8, 2014 by CTSchmidt Quote Link to post Share on other sites
Bot-Factory 602 Posted August 8, 2014 Report Share Posted August 8, 2014 Sorry i did a mistake, I meant: I have a LargeList with ID's (eg. 001, 002, 008) now i want to add a row from LargeTableA (with a matching id) to LargeTableB the result will be: LargeTableB: 001,data,data 002,data,data 008,data,dataThis was what I meant with "specific row" Thanks in advance CTSchmidt Well it's the same approach as the example I posted. You just have to apply the right logic to make it work. But the commands are the same.Loop through the stuff you have. Read the data you need, and then add it at the position where you want it. There is no other way to do that. Dan Quote Link to post Share on other sites
CTSchmidt 1 Posted August 8, 2014 Report Share Posted August 8, 2014 okay I will try it, I will post my solution btw great Stuff CTSchmidt Quote Link to post Share on other sites
UBotBuddy 331 Posted August 8, 2014 Report Share Posted August 8, 2014 While I rarely use Threads I love the fact that this reduces my overall memory footprint. VERY cool Kev. I did think of an enhancement if you decide to use it. Actually two. 1. Provide a function like "Next Large List Item" like "$next list item". That way we don't have to seed a variable in the "Large List Item". 2. Provide a way in suggestion #1 & your "Random Large List Item" to remove the item out of the List when it is selected by the command. Hmmm Just thought of a question. How do I save a Large List to a file? Thanks again! Buddy Quote Link to post Share on other sites
Bot-Factory 602 Posted August 8, 2014 Report Share Posted August 8, 2014 While I rarely use Threads I love the fact that this reduces my overall memory footprint. VERY cool Kev. I did think of an enhancement if you decide to use it. Actually two. 1. Provide a function like "Next Large List Item" like "$next list item". That way we don't have to seed a variable in the "Large List Item". 2. Provide a way in suggestion #1 & your "Random Large List Item" to remove the item out of the List when it is selected by the command. Hmmm Just thought of a question. How do I save a Large List to a file? Thanks again! Buddy save to file("d:\\x.csv", $plugin function("Bigtable.dll", "Large list return", "test")) But I think kev works on a save large list to file command as well. Dan Quote Link to post Share on other sites
UBotBuddy 331 Posted August 8, 2014 Report Share Posted August 8, 2014 Thanks Dan! 1 Quote Link to post Share on other sites
Bot-Factory 602 Posted August 9, 2014 Report Share Posted August 9, 2014 HowTo Breakdown a Large List into smaller Chunks plugin command("Bigtable.dll", "Clear large list", "testlist")plugin command("Bigtable.dll", "Clear large list", "tmplist")loop(20133) { plugin command("Bigtable.dll", "Add item to large list", "testlist", $random text(5))}alert($plugin function("Bigtable.dll", "Large list total", "testlist"))set(#listcounter1, $plugin function("Bigtable.dll", "Large list total", "testlist"), "Global")set(#listcounter2, 0, "Global")loop while($comparison(#listcounter1, ">", 0)) { if($comparison(#listcounter1, ">=", 5000)) { then { loop(5000) { plugin command("Bigtable.dll", "Add item to large list", "tmplist", $plugin function("Bigtable.dll", "Large list item", "testlist", #listcounter2)) increment(#listcounter2) } set(#listcounter1, $subtract(#listcounter1, 5000), "Global") } else { loop(#listcounter1) { plugin command("Bigtable.dll", "Add item to large list", "tmplist", $plugin function("Bigtable.dll", "Large list item", "testlist", #listcounter2)) increment(#listcounter2) } set(#listcounter1, $subtract(#listcounter1, #listcounter1), "Global") } } alert($plugin function("Bigtable.dll", "Large list total", "tmplist")) plugin command("Bigtable.dll", "Clear large list", "tmplist")} You now could use the tmplist to launch your threads. This would allow you to scrape through a list of urls for example, but save the results in between. So if you initial list has 1 million entries, they are not executed in one go. If you want to use this within threads, don't forget to change the Global variables to local ones! If someone has an idea how to simplify / optimize that, please share. CheersDan Quote Link to post Share on other sites
kev123 132 Posted August 9, 2014 Author Report Share Posted August 9, 2014 dan you could use my advanced file plugin and read in in chunks. if your reading from file Quote Link to post Share on other sites
Bot-Factory 602 Posted August 9, 2014 Report Share Posted August 9, 2014 dan you could use my advanced file plugin and read in in chunks. You mean saving the 2million urls to a tmp file. And then use "read part file" to load chunks of it?Do you think that's faster than the loop thing? Dan Quote Link to post Share on other sites
kev123 132 Posted August 9, 2014 Author Report Share Posted August 9, 2014 You mean saving the 2million urls to a tmp file. And then use "read part file" to load chunks of it?Do you think that's faster than the loop thing? DanI edit my above comment before seeing your response. If your reading from file yes if this are already in memory no keep as is. 1 Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.