Bot-Factory 602 Posted April 29, 2014 Report Share Posted April 29, 2014 Would it be possible to implement a list function as well?With add item to list? And making it thread safe? So that the example code from this thread here:http://www.ubotstudio.com/forum/index.php?/topic/15441-free-plugin-threads-counter-ubot-v4-threading-fixed/page-6 Works? Kindest regardsDan Quote Link to post Share on other sites
kev123 132 Posted April 29, 2014 Author Report Share Posted April 29, 2014 I will probably add lists at a late date but what you are asking can be done with the plugin now the command "large table bulk input " used below is thread safe. You just need to set the tables size to one column. Also for people reading ubot lists are thread safe for using say in local there not when trying to write to from several threads. I will be upgrading a couple of the commands this week to make them more efficient and adding "remove duplicates from large table" and "add large table to large table" which in effect allows you to add local tables to a main table with the use of a other plugin See example code below for solving above problem, plugin command("Bigtable.dll", "Delete Single Large Table", "bulkinputtest") comment("set max threads and reset used") set(#max, 20, "Global") set(#used, 0, "Global") divider set(#th_id, 0, "Global") loop(20) { comment("loop while max threads") loop while($comparison(#used, ">=", #max)) { wait(0.25) } divider increment(#used) returning_to_table_from_thread(#th_id) increment(#th_id) } comment("wait for threads to finish") loop while($comparison(#used, ">", 0)) { wait(0.25) } alert($plugin function("Bigtable.dll", "Return Large Table CSV", "bulkinputtest", 0, 0)) define returning_to_table_from_thread(#thread_id) { thread { set(#innerloop, 0, "Local") clear list(%somethingscraped) loop(2) { add item to list(%somethingscraped, "thread:{#thread_id}:{#innerloop}", "Don\'t Delete", "Local") increment(#innerloop) } plugin command("Bigtable.dll", "Large Table bulk input", "bulkinputtest", %somethingscraped) decrement(#used) } } Quote Link to post Share on other sites
Bot-Factory 602 Posted May 17, 2014 Report Share Posted May 17, 2014 I will probably add lists at a late date but what you are asking can be done with the plugin now the command "large table bulk input " used below is thread safe. You just need to set the tables size to one column. Also for people reading ubot lists are thread safe for using say in local there not when trying to write to from several threads. I will be upgrading a couple of the commands this week to make them more efficient and adding "remove duplicates from large table" and "add large table to large table" which in effect allows you to add local tables to a main table with the use of a other plugin See example code below for solving above problem, plugin command("Bigtable.dll", "Delete Single Large Table", "bulkinputtest") comment("set max threads and reset used") set(#max, 20, "Global") set(#used, 0, "Global") divider set(#th_id, 0, "Global") loop(20) { comment("loop while max threads") loop while($comparison(#used, ">=", #max)) { wait(0.25) } divider increment(#used) returning_to_table_from_thread(#th_id) increment(#th_id) } comment("wait for threads to finish") loop while($comparison(#used, ">", 0)) { wait(0.25) } alert($plugin function("Bigtable.dll", "Return Large Table CSV", "bulkinputtest", 0, 0)) define returning_to_table_from_thread(#thread_id) { thread { set(#innerloop, 0, "Local") clear list(%somethingscraped) loop(2) { add item to list(%somethingscraped, "thread:{#thread_id}:{#innerloop}", "Don\'t Delete", "Local") increment(#innerloop) } plugin command("Bigtable.dll", "Large Table bulk input", "bulkinputtest", %somethingscraped) decrement(#used) } } Hi Kev, I currently start using your plugin for my scrapers. Adding to lists from threads doesn't work. I'm getting lots of duplicates. And normal tables don't work if I scrape more than one thing within a thread. So currently your plugins seems to be the only solution. Are you still working on the update? ThanksDan Quote Link to post Share on other sites
kev123 132 Posted May 18, 2014 Author Report Share Posted May 18, 2014 yep commands mentioned are done I just need to tidy up some of the existing commands to make them more efficient. should have time this week. just to confirm the issues you mentioned are they with this plugin or ubots normal commands. If you have had issues with this plugin please PM me the details so I can take a look. Quote Link to post Share on other sites
Bot-Factory 602 Posted May 18, 2014 Report Share Posted May 18, 2014 No issues with your plugin kev. The only issues I have are all with ubots default features. I just did another threading test with your plugin. 100 threads. Adding 50000 items..not a single issue! No memory increases. No frozen UI. Exactly 50000 entries in the table. Amazing. I don't know how you have done it, but this is exactly how ubot lists and tables should work out of the box. Would be great if you could explain all the commands a little bit in the future. Just one sentence per command.I'm a little bit confused with the Create large table command. When do I need that exactly? Because commands like large table bulk import seem to work without it. Dan Quote Link to post Share on other sites
kev123 132 Posted May 18, 2014 Author Report Share Posted May 18, 2014 i'll post a video when I get chance pretty busy at the min. A lot of the commands where concepts and added at the last min hence lack of documentation. The create large table command is for when you create a table from fresh and then write into it using set large table cell without using commands like bulk add. I don't know how you have done it, but this is exactly how ubot lists and tables should work out of the box. Dan haven't done anything special if anything its under optimised at the min which i'll improve but average users wont notice a difference. I suspect ubot tables rebuild for every set table cell due to them auto resizing and debugger integration etc. Quote Link to post Share on other sites
Bot-Factory 602 Posted May 18, 2014 Report Share Posted May 18, 2014 It's seems I still have to initialize the large table:plugin command("Bigtable.dll", "Create Large Table", "test", 0, 0) Because if I don't do that, I get an error in the Ubot UI when I run my script the first time. If I run it again, it works fine without any error.If I "initialize" the table as mentioned above that doesn't happen at all. There is one small thing I have noticed. In my test script the performance goes down when I come closer to 50000 entries. Not sure if that is related to Large Table bulk import or something else. Test Script:plugin command("Bigtable.dll", "Clear all Large Tables")plugin command("Bigtable.dll", "Create Large Table", "test", 0, 0)set(#threadcount, $plugin function("Threads Counter.dll", "threads counter", "reset"), "Global")set(#Loops, 50000, "Global")ui stat monitor("Open Threads:", #threadcount)ui stat monitor("Total:", #totalentries)set(#loopcounter, 0, "Global")loop(#Loops) { loop while($comparison($plugin function("Threads Counter.dll", "threads counter", "read"), ">=", 100)) { set(#threadcount, $plugin function("Threads Counter.dll", "threads counter", "read"), "Global") wait(0.1) } set(#threadcount, $plugin function("Threads Counter.dll", "threads counter", "increment"), "Global") testthreads4($random text(50)) increment(#loopcounter) set(#totalentries, $plugin function("Bigtable.dll", "Large Table Total Rows", "test"), "Global")}loop while($comparison(#threadcount, ">", 0)) { set(#threadcount, $plugin function("Threads Counter.dll", "threads counter", "read"), "Global") wait(0.05)}set(#threadcount, $plugin function("Threads Counter.dll", "threads counter", "read"), "Global")set(#totalentries, $plugin function("Bigtable.dll", "Large Table Total Rows", "test"), "Global")alert($plugin function("Bigtable.dll", "Large Table Total Rows", "test"))stop scriptdefine testthreads4(#number) { thread { plugin command("Bigtable.dll", "Large Table bulk input", "test", #number) set(#threadcount, $plugin function("Threads Counter.dll", "threads counter", "decrement"), "Global") }} CheersDan Quote Link to post Share on other sites
Bot-Factory 602 Posted May 19, 2014 Report Share Posted May 19, 2014 Issue: The command:plugin command("Bigtable.dll", "Create Large Table", "Test", 0, 0) will populate the large table with old data. Example:plugin command("Bigtable.dll", "Clear all Large Tables")plugin command("Bigtable.dll", "Create Large Table", "Name", 0, 0)plugin command("Bigtable.dll", "Large Table bulk input", "Name", "Content")alert($plugin function("Bigtable.dll", "Return Large Table CSV", "Name", 0, 0))plugin command("Bigtable.dll", "Large Table bulk input", "Name", "Content2")plugin command("Bigtable.dll", "Clear all Large Tables")plugin command("Bigtable.dll", "Create Large Table", "Name", 0, 0)alert($plugin function("Bigtable.dll", "Return Large Table CSV", "Name", 0, 0)) Dan Quote Link to post Share on other sites
kev123 132 Posted May 19, 2014 Author Report Share Posted May 19, 2014 Thanks dan ill check it out. Regarding the bulk input the issue is fixed in the version I'm using I'll also do a example for you single adding isnt the best way to use that command. 1 Quote Link to post Share on other sites
Marq 0 Posted May 19, 2014 Report Share Posted May 19, 2014 This plugin is amazing. I build a bot that navigated through a list of over 100K items, and it would take almost 3-4 minutes before the list would even load and eventually would cause crashes or freezing in Ubot as the Browser ran. I changed the list to a big table using this plugin and the data loads almost instantly and Ubot is running smooth like a champ. This is seriously a must-have plugin when dealing with large amounts of data. Thanks! Quote Link to post Share on other sites
Bot-Factory 602 Posted May 20, 2014 Report Share Posted May 20, 2014 Small Glitch? Commands like:Large Table Total RowsLarge Table Total Columns Return an error in the UI when the large table is empty. Would be great if it would return 0 in that case. ThanksDan Quote Link to post Share on other sites
Bot-Factory 602 Posted May 22, 2014 Report Share Posted May 22, 2014 Would it be possible to useplugin command("Bigtable.dll", "Set Large Table cell", "tmp", 0, 0, "asd") without the need to create a table first? Because sometimes I don't know how many entries I will need in a table.And bulk import seems to slow down a lot over time. So I though about using a counter and:plugin command("Bigtable.dll", "Set Large Table cell", "tmp", #rownumber, 0, #data) Or what would you recommend? Dan Quote Link to post Share on other sites
kev123 132 Posted May 22, 2014 Author Report Share Posted May 22, 2014 i'm going to do a video when I next get chance bulk input should really only be used for inputting a bunch of data in one and not single entries say for example lists with a hundred entries. The reason for this is it has to rebuild the table every time it is used. I was toying with the idea of not having to define the table size when your using it without other commands like you currently do.the reason i'm against this is because within the plugin code I would have to rebuild the table to auto resize unless I created lists of lists. The table with just the set table cell basic commands was intended to be super lightweight. If you run it with a few hundred threads setting several cells in each thread the resource use is very low, that's its main purpose. I think if I was to add lists this would solve most peoples needs where there adding single items. i'll add it to my list of things to do. 1 Quote Link to post Share on other sites
Bot-Factory 602 Posted May 22, 2014 Report Share Posted May 22, 2014 Awesome. Thanks a lot kev. I mainly need lists as well. But every thread could have between 1 and hundred results for example. It would be great if I could do add to large list from text or add list from list kind of things but thread safe and lightweight.Remove duplicates would be also nice. But that's probably to much overhead. Dan Quote Link to post Share on other sites
kev123 132 Posted May 22, 2014 Author Report Share Posted May 22, 2014 Awesome. Thanks a lot kev. I mainly need lists as well. But every thread could have between 1 and hundred results for example. It would be great if I could do add to large list from text or add list from list kind of things but thread safe and lightweight.Remove duplicates would be also nice. But that's probably to much overhead. Danall put on the plan basic stuff with lists Quote Link to post Share on other sites
Bot-Factory 602 Posted May 22, 2014 Report Share Posted May 22, 2014 Awesome! Sad that you have to do all the hard work for stuff that should work out of the box... Dan Quote Link to post Share on other sites
kev123 132 Posted June 12, 2014 Author Report Share Posted June 12, 2014 update 1.1 check your email fixfixed a issue with tables not being created first.Fixed a bug where old data was returned with the return large table csv commandfixed a issue where using return csv and bulk input caused a formatting error.improvedthe large table from file command will now allow "," etc in cellstidy several of the commandsAddedcommandsRemove duplicates large tableCombine large tableslarge table to filefunction"thread id" you can make tables local using this( not sure if in other plugin already has this but couldn't find so added) Quote Link to post Share on other sites
Bot-Factory 602 Posted June 13, 2014 Report Share Posted June 13, 2014 Hi. Thanks a lot for update 1.1 Not sure why but I get a small error:plugin command("Bigtable.dll", "Clear all Large Tables")plugin command("Bigtable.dll", "Large Table bulk input", "Name", "hans,peter")plugin command("Bigtable.dll", "Large Table bulk input", "Name", "hans")plugin command("Bigtable.dll", "Large Table bulk input", "Name", "Content2,xxxxasdasd,xxxx")alert($plugin function("Bigtable.dll", "Return Large Table CSV", "Name", 0, 0))plugin command("Bigtable.dll", "Large Table Save file", "d:\\x.csv", "Name") The bulk input creates a strange format. There are to many , in the table?And save to file is not working when the format of the table is messed up. CheersDan Quote Link to post Share on other sites
kev123 132 Posted June 13, 2014 Author Report Share Posted June 13, 2014 Ok strange I will check worked ok at my end Quote Link to post Share on other sites
kev123 132 Posted June 13, 2014 Author Report Share Posted June 13, 2014 update 1.2(check your emails) fixed a small issue with bulk inputfixed a issue where null values caused save to file to error 1 Quote Link to post Share on other sites
Bot-Factory 602 Posted June 13, 2014 Report Share Posted June 13, 2014 Thanks for the 1.2 fix. Unfortunately I found another issue. Support for Multi Line Items. When I have a multi line table cell, excel for example adds a "" around it: cell1,"firstlinesecond line",cell3 But the plugin doesn't put the "" around a multi line item. And when I add "" myself, they get duplicated.plugin command("Bigtable.dll", "Clear all Large Tables")plugin command("Bigtable.dll", "Create Large Table", "test", 1, 2)plugin command("Bigtable.dll", "Set Large Table cell", "test", 0, 0, "First LineSecond Line")alert($plugin function("Bigtable.dll", "Return Large Table CSV", "test", 0, 1))save to file("d:\\x.csv", $plugin function("Bigtable.dll", "Return Large Table CSV", "test", 0, 1)) Same happens with Bulk Import. plugin command("Bigtable.dll", "Clear all Large Tables")plugin command("Bigtable.dll", "Large Table bulk input", "Name", "Content2,xxxxasdasd,xxxx")alert($plugin function("Bigtable.dll", "Return Large Table CSV", "Name", 0, 0))plugin command("Bigtable.dll", "Large Table Save file", "d:\\x.csv", "Name") Not sure how bulk import should handle that. If it's pure CSV format, it normally should support something like:plugin command("Bigtable.dll", "Clear all Large Tables")plugin command("Bigtable.dll", "Large Table bulk input", "Name", "Content2,\"xxxxasdasd\",xxxx")alert($plugin function("Bigtable.dll", "Return Large Table CSV", "Name", 0, 0))plugin command("Bigtable.dll", "Large Table Save file", "d:\\x.csv", "Name") But then I get:Content2,"""xxxx""asdasd""",xxxx in the csv. Just as a little hint. A multi line item could also contain a " as part of the text. So that should be handled correctly as well. Dan Quote Link to post Share on other sites
kev123 132 Posted June 14, 2014 Author Report Share Posted June 14, 2014 bulk import, imports on 1,2,3,4,5,6,7 basically a simple comma between fields it is not a csv importer. Imagine people who want to a simple import having to edit there fields making CSV compliant not really suitable. Same for return large table csv mainly for debugging below commands are better suited. The reason save to file and read for file where added was because they where built for the correct formatted csv file as you have described.If there's a problem with these two commands from a csv format point of view please let me know. please use save to file or read for proper csv format. thanksKev Quote Link to post Share on other sites
Bot-Factory 602 Posted June 14, 2014 Report Share Posted June 14, 2014 bulk import, imports on 1,2,3,4,5,6,7 basically a simple comma between fields it is not a csv importer. Imagine people who want to a simple import having to edit there fields making CSV compliant not really suitable. Same for return large table csv mainly for debugging below commands are better suited. The reason save to file and read for file where added was because they where built for the correct formatted csv file as you have described.If there's a problem with these two commands from a csv format point of view please let me know. please use save to file or read for proper csv format. thanksKevAh ok. Now I understand. It's all in the details :-) Great work Kev Dan Quote Link to post Share on other sites
kev123 132 Posted June 14, 2014 Author Report Share Posted June 14, 2014 Ah ok. Now I understand. It's all in the details :-) Great work Kev Dan also check out the add table to table its a more efficent way of combining tables than the example via pm we discussed. With the thread id function and a bit of smart programming you can have local tables adding to a master table hope this helps Quote Link to post Share on other sites
kev123 132 Posted June 26, 2014 Author Report Share Posted June 26, 2014 updating the large data plugin apart from the functions already in ubot for lists is there anything else you would like for list functions and commands Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.