[Free] Plugin Large Data

[Bot-Factory 602]

List to Table Row

List to Table column 

 

With Start Row / Start Column / append  - Options

 

would be nice. 

 

Can you tell when the table features will be available? Working on a new bot where this would be very handy.

 

Thanks

Dan

[kev123 132]

Dan the list update is ready to go on travel laptop at mo so as soon as I get home I'll upload. Theres a good few commands and funcrions, I put a few million in a list with great ease when testing.

 

As a bonus if the last few tests go to plan when I get home i should have parts of thw local dictionary plugin added as well so basically you can add a local list/variable to the large table/list. Credit to aymen for his involvement.

[Sirfrank 17]

buddy, this took my ubot to  a whole new level, thanks for it, and a big thumbs up for keeping it free.

[Bot-Factory 602]

Dan the list update is ready to go on travel laptop at mo so as soon as I get home I'll upload. Theres a good few commands and funcrions, I put a few million in a list with great ease when testing.

 

As a bonus if the last few tests go to plan when I get home i should have parts of thw local dictionary plugin added as well so basically you can add a local list/variable to the large table/list. Credit to aymen for his involvement.

Awesome! Can't wait :-)

 

Dan

[Bot-Factory 602]

Kev,

 

 

The Random List Item Function  would be awesome as well.

 

Dan

[Bot-Factory 602]

Would it be possible to add a local thread safe variable as well? 

With add variable and get variable?

 

UPDATE: Ok, I can use a table with thread id and 1 row 1 column to do that. So that works already. 

 

 

Thanks

Dan

[Bot-Factory 602]

What is:

• Remove duplicates large table 

doing exactly?

 

I have a large table with 500 duplicate entries. But when I execute the command it gives me an error.

"The object reference was not set to an object instance"   (Translated from German. So the English message might be slightly different)

 

Dan

[Bot-Factory 602]

Feature Request:

 

Remove empty rows.

 

I have a large table with 500 entries. And I want to remove all rows where the Status of 1 Column=Error.

So that I have a new table with maybe 350 entries.

 

The challenge is that I have to create the large tables upfront. So my second table has the same amount of rows than my first table. 

I could work with Bulk Input. But that's probably not the right way to do it. 

 

What's the smartest way to archive this? Without using ubot lists. 

 

 

Here's the code I have so far:

plugin command("Bigtable.dll", "Clear all Large Tables")
plugin command("Bigtable.dll", "Ubot Table to Large Table", &tmptable, "tmp")
plugin command("Bigtable.dll", "Create Large Table", "tmp2", $plugin function("Bigtable.dll", "Large Table Total Rows", "tmp"), $plugin function("Bigtable.dll", "Large Table Total Columns", "tmp"))
set(#loopcounter, 0, "Global")
set(#loopcounter2, 0, "Global")
loop($plugin function("Bigtable.dll", "Large Table Total Rows", "tmp")) {
    if($comparison($plugin function("Bigtable.dll", "Large Table Cell", "tmp", #loopcounter, 2), "=", "EmailFehler")) {
        then {
        }
        else {
            plugin command("Bigtable.dll", "Set Large Table cell", "tmp2", #loopcounter2, 0, $plugin function("Bigtable.dll", "Large Table Cell", "tmp", #loopcounter, 0))
            plugin command("Bigtable.dll", "Set Large Table cell", "tmp2", #loopcounter2, 1, $plugin function("Bigtable.dll", "Large Table Cell", "tmp", #loopcounter, 1))
            plugin command("Bigtable.dll", "Set Large Table cell", "tmp2", #loopcounter2, 2, $plugin function("Bigtable.dll", "Large Table Cell", "tmp", #loopcounter, 2))
            increment(#loopcounter2)
        }
    }
    increment(#loopcounter)
}
clear table(&tmptable2)
alert($plugin function("Bigtable.dll", "Large Table Total Rows", "tmp"))
alert($plugin function("Bigtable.dll", "Large Table Total Rows", "tmp2"))
plugin command("Bigtable.dll", "Return Large Table", &tmptable2, "tmp2", 0, $subtract($plugin function("Bigtable.dll", "Large Table Total Rows", "tmp2"), 1))

 

 

Some other stuff that might be useful:

Add to List from Column
Add to List from Row
Delete Column
Delete Row
Sort Table by Row
Merge Tables
 

 

 

Cheers

Dan

[kev123 132]

Dan theres already a combine tables command. I will add a remove empty rows command. Your other suggestions ill review and maybe add in a latter update

[kev123 132]

also the remove duplicates should do what it says if your having problems please pm me a example script.

[Bot-Factory 602]

also the remove duplicates should do what it says if your having problems please pm me a example script.

The error happens when not all columns have data. So if I create a table with 2 columns for example but only fill 1. 

 

Example code:

plugin command("Bigtable.dll", "Clear all Large Tables")

plugin command("Bigtable.dll", "Create Large Table", "tmp", 50, 2)

set(#loopcounter, 0, "Global")

loop(50) {

    plugin command("Bigtable.dll", "Set Large Table cell", "tmp", #loopcounter, 0, " ")

    increment(#loopcounter)

}

plugin command("Bigtable.dll", "Remove duplicate large table", "tmp")

alert($plugin function("Bigtable.dll", "Large Table Total Rows", "tmp"))

 

 

 

If I also fill the second column it works fine:

 

plugin command("Bigtable.dll", "Clear all Large Tables")

plugin command("Bigtable.dll", "Create Large Table", "tmp", 50, 2)

set(#loopcounter, 0, "Global")

loop(50) {

    plugin command("Bigtable.dll", "Set Large Table cell", "tmp", #loopcounter, 0, " ")

    plugin command("Bigtable.dll", "Set Large Table cell", "tmp", #loopcounter, 1, " ")

    increment(#loopcounter)

}

plugin command("Bigtable.dll", "Remove duplicate large table", "tmp")

alert($plugin function("Bigtable.dll", "Large Table Total Rows", "tmp"))

 

 

Dan

[kev123 132]

The error happens when not all columns have data. So if I create a table with 2 columns for example but only fill 1. 

 

Example code:

plugin command("Bigtable.dll", "Clear all Large Tables")

plugin command("Bigtable.dll", "Create Large Table", "tmp", 50, 2)

set(#loopcounter, 0, "Global")

loop(50) {

    plugin command("Bigtable.dll", "Set Large Table cell", "tmp", #loopcounter, 0, " ")

    increment(#loopcounter)

}

plugin command("Bigtable.dll", "Remove duplicate large table", "tmp")

alert($plugin function("Bigtable.dll", "Large Table Total Rows", "tmp"))

 

 

 

If I also fill the second column it works fine:

 

plugin command("Bigtable.dll", "Clear all Large Tables")

plugin command("Bigtable.dll", "Create Large Table", "tmp", 50, 2)

set(#loopcounter, 0, "Global")

loop(50) {

    plugin command("Bigtable.dll", "Set Large Table cell", "tmp", #loopcounter, 0, " ")

    plugin command("Bigtable.dll", "Set Large Table cell", "tmp", #loopcounter, 1, " ")

    increment(#loopcounter)

}

plugin command("Bigtable.dll", "Remove duplicate large table", "tmp")

alert($plugin function("Bigtable.dll", "Large Table Total Rows", "tmp"))

 

 

Dan

ok thanks dan as always.

I know what the cause is I will put in the update after next, after thinking about it this command is over complex so might rewrite.

[Bot-Factory 602]

Kev, I have a quick question.

 

When I'm using the plugin with multithreading. And each thread scrapes like 5-30 urls. 

And at the end all urls from all threads should be in one single table/list and I want to remove duplicates.

 

What's the best and most efficient way to do that? Talking about a total of 1million urls at the end. 

Bulk import and remove duplicate large table?

 

Thanks in advance for your help.

Dan

[kev123 132]

at the end of each thread use the Combine large tables command and remove duplicates or wait 2 secs and i'll send the update out which includes lists

[Bot-Factory 602]

1.....2......   

 

Thank's a lot Kev! You are doing a great job! I probably would have switched to C# without all the great plugins...

 

Dan

[kev123 132]

commands
Large table Remove empty
add item to large list
ubot list to large list
clear list
Add Large list to list
compare large lists
large list from file
Large list Remove duplicates
Sort large list

 

functions
return large list
Large list insert
Large list item
Large list total
Random Large list item

 

couldn't add intergration with local dic plugin in this update my method didn't work i'll look for a fix and maybe ask aymen.

people who have previously downloaded check your email.

[Bot-Factory 602]

Awesome!! Thanks a lot Kev.

 

What do you mean by "intergration with local dic plugin"?

Does that mean they are not working together?

Dan
 

[kev123 132]

Awesome!! Thanks a lot Kev.

 

What do you mean by "intergration with local dic plugin"?

Does that mean they are not working together?

Dan

 

sorry for the confusion I thought I  talked about this. Basically so you can pass data from local dic to large data plugin. I haven't added these commands as there was a small issue.

 

As things are now there is no issue and both plugins work as they are separately.

[kev123 132]

ps find the biggest list you can and read from file I think you will be very happy

[Bot-Factory 602]

Great Job Kev.

 

Adding 500000 entries to a large list took 26seconds. Ubot memory increased to 1.7GB, but was released 4 seconds after the operation was finished. Then it was back down to 200Mb. 

Now slowdown of the ubot UI at all. 

 

So this is even faster than large table. Adding 500k entries to large table took 44seconds on my system. 

 

Thank you kev!

 

Dan

[Bot-Factory 602]

Feature Request:

 

Large List from text with Delimiter support. 

 

Similar to the following ubot command:

add list to list(%tmp, $list from text("test

hallo
welt", $new line), "Delete", "Global")

 

Thanks in advance

Dan

[kev123 132]

Great Job Kev.

 

Adding 500000 entries to a large list took 26seconds. Ubot memory increased to 1.7GB, but was released 4 seconds after the operation was finished. Then it was back down to 200Mb. 

Now slowdown of the ubot UI at all. 

 

So this is even faster than large table. Adding 500k entries to large table took 44seconds on my system. 

 

Thank you kev!

 

Dan

just to clarify that was looping or something? not reading from file ? if its reading from file I have some serious fixing to do.

[Bot-Factory 602]

just to clarify that was looping or something? not reading from file ? if its reading from file I have some serious fixing to do.

 

loop(500000) {

}

 

Dan

[Bot-Factory 602]

Kev,

 

from a performance point of view, is it ok to do:

ui stat monitor("XX", $plugin function("Bigtable.dll", "Large list total", "test"))

 

Or should I use a variable for that?

 

Not sure how often ui stat monitor calls the plugin function?

 

Dan

[Bot-Factory 602]

Feature Request:

 

 

1.)

Support for local ubot lists with the following command:

plugin command("Bigtable.dll", "ubot list to large list", "scrapedata", %scrapedata)

 

If I call that right after I store stuff in a  local ubot list:

add list to list(%scrapedata, $list from text($find regular expression($document text, #scraperegex), $new line), "Delete", "Local")

 

It doesn't work

 

2.)

Large list remove empty. Similar to large table remove empty

 

Dan