Jump to content
UBot Underground

[Free] Plugin Large Data


Recommended Posts

Would it be possible to implement a list function as well?

With add item to list?

 

And making it thread safe? So that the example code from this thread here:

http://www.ubotstudio.com/forum/index.php?/topic/15441-free-plugin-threads-counter-ubot-v4-threading-fixed/page-6

 

Works?

 

Kindest regards

Dan

Link to post
Share on other sites
  • Replies 209
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

My latest free plugin good for large table data storing   commands and functions are   http://robobest.com/wp-content/uploads/2014/04/bigdatacommands.png   http://robobest.com/wp-content/uploads/2014/

I edit my above comment before seeing your response. If your reading from file yes if this are already in memory no keep as is.

Much Appreciated Work KEV downloaded and will try now   

I will probably add lists at a late date but what you are asking can be done with the plugin now the command "large table bulk input " used below is thread safe. You just need to set the tables size to one column.

 

Also for people reading ubot lists are thread safe for using say in local there not when trying to write to from several threads.

 

 I will be upgrading a couple of the commands this week to make them more efficient and adding "remove duplicates from large table" and "add large table to large table" which in effect allows you to add local tables to a main table with the use of a other plugin

 

See example code below for solving above problem,

plugin command("Bigtable.dll", "Delete Single Large Table", "bulkinputtest")
comment("set max threads and reset used")
set(#max, 20, "Global")
set(#used, 0, "Global")
divider
set(#th_id, 0, "Global")
loop(20) {
    comment("loop while max threads")
    loop while($comparison(#used, ">=", #max)) {
        wait(0.25)
    }
    divider
    increment(#used)
    returning_to_table_from_thread(#th_id)
    increment(#th_id)
}
comment("wait for threads to finish")
loop while($comparison(#used, ">", 0)) {
    wait(0.25)
}
alert($plugin function("Bigtable.dll", "Return Large Table CSV", "bulkinputtest", 0, 0))
define returning_to_table_from_thread(#thread_id) {
    thread {
        set(#innerloop, 0, "Local")
        clear list(%somethingscraped)
        loop(2) {
            add item to list(%somethingscraped, "thread:{#thread_id}:{#innerloop}", "Don\'t Delete", "Local")
            increment(#innerloop)
        }
        plugin command("Bigtable.dll", "Large Table bulk input", "bulkinputtest", %somethingscraped)
        decrement(#used)
    }
}

Link to post
Share on other sites
  • 3 weeks later...

 

I will probably add lists at a late date but what you are asking can be done with the plugin now the command "large table bulk input " used below is thread safe. You just need to set the tables size to one column.

 

Also for people reading ubot lists are thread safe for using say in local there not when trying to write to from several threads.

 

 I will be upgrading a couple of the commands this week to make them more efficient and adding "remove duplicates from large table" and "add large table to large table" which in effect allows you to add local tables to a main table with the use of a other plugin

 

See example code below for solving above problem,

plugin command("Bigtable.dll", "Delete Single Large Table", "bulkinputtest")
comment("set max threads and reset used")
set(#max, 20, "Global")
set(#used, 0, "Global")
divider
set(#th_id, 0, "Global")
loop(20) {
    comment("loop while max threads")
    loop while($comparison(#used, ">=", #max)) {
        wait(0.25)
    }
    divider
    increment(#used)
    returning_to_table_from_thread(#th_id)
    increment(#th_id)
}
comment("wait for threads to finish")
loop while($comparison(#used, ">", 0)) {
    wait(0.25)
}
alert($plugin function("Bigtable.dll", "Return Large Table CSV", "bulkinputtest", 0, 0))
define returning_to_table_from_thread(#thread_id) {
    thread {
        set(#innerloop, 0, "Local")
        clear list(%somethingscraped)
        loop(2) {
            add item to list(%somethingscraped, "thread:{#thread_id}:{#innerloop}", "Don\'t Delete", "Local")
            increment(#innerloop)
        }
        plugin command("Bigtable.dll", "Large Table bulk input", "bulkinputtest", %somethingscraped)
        decrement(#used)
    }
}

Hi Kev,

 

I currently start using your plugin for my scrapers. Adding to lists from threads doesn't work. I'm getting lots of duplicates. 

And normal tables don't work if I scrape more than one thing within a thread.

 

So currently your plugins seems to be the only solution.

 

Are you still working on the update?

 

 

Thanks

Dan

Link to post
Share on other sites

yep commands mentioned are done I just need to tidy up some of the existing commands to make them more efficient. should have time this week. just to confirm the issues you mentioned are they with this plugin or ubots normal commands. If you have had issues with this plugin please PM me the details so I can take a look.

Link to post
Share on other sites

No issues with your plugin kev. The only issues I have are all with ubots default features. 

 

 

I just did another threading test with your plugin. 100 threads. Adding 50000 items..

not a single issue! No memory increases. No frozen UI. Exactly 50000 entries in the table. 

 

Amazing. 

 

I don't know how you have done it, but this is exactly how ubot lists and tables should work out of the box. 

 

 

Would be great if you could explain all the commands a little bit in the future. Just one sentence per command.

I'm a little bit confused with the Create large table command. When do I need that exactly? Because commands like large table bulk import seem to work without it.

 

 

 

Dan

Link to post
Share on other sites

i'll post a video when I get chance pretty busy at the min. A lot of the commands where concepts and added at the last min hence lack of documentation. The create large table command is for when you create a table from fresh and then write into it using set large table cell without using commands like bulk add.

 

 

 I don't know how you have done it, but this is exactly how ubot lists and tables should work out of the box. 

 

Dan

 

haven't done anything special if anything its under optimised at the min which i'll improve but average users wont notice a difference. I suspect ubot tables rebuild for every set table cell due to them auto resizing and debugger integration etc.

Link to post
Share on other sites

It's seems I still have to initialize the large table:

plugin command("Bigtable.dll""Create Large Table""test", 0, 0)

 

Because if I don't do that, I get an error in the Ubot UI when I run my script the first time. If I run it again, it works fine without any error.

If I "initialize" the table as mentioned above that doesn't happen at all.

 

 

There is one small thing I have noticed. In my test script the performance goes down when I come closer to 50000 entries. 

Not sure if that is related to Large Table bulk import or something else.

 

 

Test Script:

plugin command("Bigtable.dll""Clear all Large Tables")
plugin command("Bigtable.dll""Create Large Table""test", 0, 0)
set(#threadcount$plugin function("Threads Counter.dll""threads counter""reset"), "Global")
set(#Loops, 50000, "Global")
ui stat monitor("Open Threads:"#threadcount)
ui stat monitor("Total:"#totalentries)
set(#loopcounter, 0, "Global")
loop(#Loops) {
    loop while($comparison($plugin function("Threads Counter.dll""threads counter""read"), ">=", 100)) {
        set(#threadcount$plugin function("Threads Counter.dll""threads counter""read"), "Global")
        wait(0.1)
    }
    set(#threadcount$plugin function("Threads Counter.dll""threads counter""increment"), "Global")
    testthreads4($random text(50))
    increment(#loopcounter)
    set(#totalentries$plugin function("Bigtable.dll""Large Table Total Rows""test"), "Global")
}
loop while($comparison(#threadcount">", 0)) {
    set(#threadcount$plugin function("Threads Counter.dll""threads counter""read"), "Global")
    wait(0.05)
}
set(#threadcount$plugin function("Threads Counter.dll""threads counter""read"), "Global")
set(#totalentries$plugin function("Bigtable.dll""Large Table Total Rows""test"), "Global")
alert($plugin function("Bigtable.dll""Large Table Total Rows""test"))
stop script
define testthreads4(#number) {
    thread {
        plugin command("Bigtable.dll""Large Table bulk input""test"#number)
        set(#threadcount$plugin function("Threads Counter.dll""threads counter""decrement"), "Global")
    }
}

 

Cheers

Dan

Link to post
Share on other sites

Issue:

 

The command:

plugin command("Bigtable.dll""Create Large Table""Test", 0, 0)

 

will populate the large table with old data.

 

 

 

Example:

plugin command("Bigtable.dll""Clear all Large Tables")
plugin command("Bigtable.dll""Create Large Table""Name", 0, 0)
plugin command("Bigtable.dll""Large Table bulk input""Name""Content")
alert($plugin function("Bigtable.dll""Return Large Table CSV""Name", 0, 0))
plugin command("Bigtable.dll""Large Table bulk input""Name""Content2")
plugin command("Bigtable.dll""Clear all Large Tables")
plugin command("Bigtable.dll""Create Large Table""Name", 0, 0)
alert($plugin function("Bigtable.dll""Return Large Table CSV""Name", 0, 0))

 

 

Dan

Link to post
Share on other sites

Thanks dan ill check it out. Regarding the bulk input the issue is fixed in the version I'm using I'll also do a example for you single adding isnt the best way to use that command.

  • Like 1
Link to post
Share on other sites

This plugin is amazing. I build a bot that navigated through a list of over 100K items, and it would take almost 3-4 minutes before the list would even load and eventually would cause crashes or freezing in Ubot as the Browser ran. I changed the list to a big table using this plugin and the data loads almost instantly and Ubot is running smooth like a champ. This is seriously a must-have plugin when dealing with large amounts of data.

 

Thanks!

Link to post
Share on other sites

Small Glitch?

 

Commands like:

Large Table Total Rows

Large Table Total Columns

 

Return an error in the UI when the large table is empty. Would be great if it would return 0 in that case.

 

Thanks

Dan

Link to post
Share on other sites

Would it be possible to use

plugin command("Bigtable.dll""Set Large Table cell""tmp", 0, 0, "asd")

 

without the need to create a table first? 

Because sometimes I don't know how many entries I will need in a table.

And bulk import seems to slow down a lot over time.

 

So I though about using a counter and:

plugin command("Bigtable.dll""Set Large Table cell""tmp"#rownumber, 0, #data)

 

Or what would you recommend?

 

Dan

Link to post
Share on other sites

i'm going to do a video when I next get chance bulk input should really only be used for inputting a bunch of data in one and not single entries say for example  lists with a hundred entries. The reason for this is it has to rebuild the table every time it is used.

 

I was toying with the idea of not having to define the table size when your using it without other commands like you currently do.

the reason i'm against this is because within the plugin code I would have to rebuild the table to auto resize unless I created lists of lists. The table with just the set table cell basic commands was intended to be super lightweight. If you run it with a few hundred threads setting several cells in each thread the resource use is very low, that's its main purpose.

 

I think if I was to add lists this would solve most peoples needs where there adding single items. i'll add it to my list of things to do.

  • Like 1
Link to post
Share on other sites

Awesome. Thanks a lot kev.

 

I mainly need lists as well. But every thread could have between 1 and hundred results for example. It would be great if I could do add to large list from text or add list from list kind of things but thread safe and lightweight.

Remove duplicates would be also nice. But that's probably to much overhead.

 

Dan

Link to post
Share on other sites

Awesome. Thanks a lot kev.

 

I mainly need lists as well. But every thread could have between 1 and hundred results for example. It would be great if I could do add to large list from text or add list from list kind of things but thread safe and lightweight.

Remove duplicates would be also nice. But that's probably to much overhead.

 

Dan

all put on the plan basic stuff with lists

Link to post
Share on other sites
  • 3 weeks later...

update 1.1 check your email

 

fix

  • fixed a issue with tables not being created first.
  • Fixed a bug where old data was returned with the return large table csv command
  • fixed a issue where using return csv and bulk input caused a formatting error.

improved

  • the large table from file command will now allow "," etc in cells
  • tidy several of the commands

Added

commands

  • Remove duplicates large table
  • Combine large tables
  • large table to file

function

  • "thread id" you can make tables local using this( not sure if in other plugin already has this but couldn't find so added)
Link to post
Share on other sites

Hi.

 

Thanks a lot for update 1.1

 

Not sure why but I get a small error:

plugin command("Bigtable.dll""Clear all Large Tables")
plugin command("Bigtable.dll""Large Table bulk input""Name""hans,peter")
plugin command("Bigtable.dll""Large Table bulk input""Name""hans")
plugin command("Bigtable.dll""Large Table bulk input""Name""Content2,xxxx
asdasd,xxxx")
alert($plugin function("Bigtable.dll""Return Large Table CSV""Name", 0, 0))
plugin command("Bigtable.dll""Large Table Save file""d:\\x.csv""Name")

 

The bulk input creates a strange format. There are to many , in the table?

And save to file is not working when the format of the table is messed up.

 

 

Cheers

Dan

Link to post
Share on other sites

Thanks for the 1.2 fix. 

 

Unfortunately I found another issue.

 

Support for Multi Line Items. When I have a multi line table cell, excel for example adds a "" around it:

 

cell1,"firstline

second line",cell3

 

But the plugin doesn't put the "" around a multi line item. And when I add "" myself, they get duplicated.

plugin command("Bigtable.dll""Clear all Large Tables")
plugin command("Bigtable.dll""Create Large Table""test", 1, 2)
plugin command("Bigtable.dll""Set Large Table cell""test", 0, 0, "First Line
Second Line")
alert($plugin function("Bigtable.dll""Return Large Table CSV""test", 0, 1))
save to file("d:\\x.csv"$plugin function("Bigtable.dll""Return Large Table CSV""test", 0, 1))

 

 

Same happens with Bulk Import.

 

plugin command("Bigtable.dll""Clear all Large Tables")
plugin command("Bigtable.dll""Large Table bulk input""Name""Content2,xxxx
asdasd,xxxx")
alert($plugin function("Bigtable.dll""Return Large Table CSV""Name", 0, 0))
plugin command("Bigtable.dll""Large Table Save file""d:\\x.csv""Name")

 

Not sure how bulk import should handle that. If it's pure CSV format, it normally should support something like:

plugin command("Bigtable.dll""Clear all Large Tables")
plugin command("Bigtable.dll""Large Table bulk input""Name""Content2,\"xxxx
asdasd\",xxxx")
alert($plugin function("Bigtable.dll""Return Large Table CSV""Name", 0, 0))
plugin command("Bigtable.dll""Large Table Save file""d:\\x.csv""Name")

 

But then I get:

Content2,"""xxxx"
"asdasd""",xxxx
 
in the csv.
 
 

Just as a little hint. A multi line item could also contain a " as part of the text. So that should be handled correctly as well.

 

 

 

Dan

Link to post
Share on other sites

bulk import, imports on 1,2,3,4,5,6,7 basically a simple comma between fields it is not a csv importer. Imagine people who want to a simple import having to edit there fields making  CSV compliant not really suitable.

 

Same for return large table csv mainly for debugging below commands are better suited.

 

The reason save to file and read for file where added was because they where built for the correct formatted csv file as you have described.

If there's a problem with these two commands from a csv format point of view please let me know.

 

please use save to file or read for proper csv format.

 

thanks

Kev

Link to post
Share on other sites

bulk import, imports on 1,2,3,4,5,6,7 basically a simple comma between fields it is not a csv importer. Imagine people who want to a simple import having to edit there fields making  CSV compliant not really suitable.

 

Same for return large table csv mainly for debugging below commands are better suited.

 

The reason save to file and read for file where added was because they where built for the correct formatted csv file as you have described.

If there's a problem with these two commands from a csv format point of view please let me know.

 

please use save to file or read for proper csv format.

 

thanks

Kev

Ah ok. Now I understand. It's all in the details :-)

 

Great work Kev

 

Dan

Link to post
Share on other sites

Ah ok. Now I understand. It's all in the details :-)

 

Great work Kev

 

Dan

also check out the add table to table its a more efficent way of combining tables than the example via pm we discussed. With the thread id function and a bit of smart programming you can have local tables adding to a master table :) hope this helps
Link to post
Share on other sites
  • 2 weeks later...

updating the large data plugin apart from the functions already in ubot for lists is there anything else you would like for list functions and commands

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...