[Free] Plugin Large Data

[kev123 132]

Feature Request:

 

 

1.)

Support for local ubot lists with the following command:

plugin command("Bigtable.dll", "ubot list to large list", "scrapedata", %scrapedata)

 

If I call that right after I store stuff in a  local ubot list:

add list to list(%scrapedata, $list from text($find regular expression($document text, #scraperegex), $new line), "Delete", "Local")

 

It doesn't work

 

2.)

Large list remove empty. Similar to large table remove empty

 

Dan

 

 

Feature Request:

 

 

1.)

Support for local ubot lists with the following command:

plugin command("Bigtable.dll", "ubot list to large list", "scrapedata", %scrapedata)

 

If I call that right after I store stuff in a  local ubot list:

add list to list(%scrapedata, $list from text($find regular expression($document text, #scraperegex), $new line), "Delete", "Local")

 

It doesn't work

 

2.)

Large list remove empty. Similar to large table remove empty

 

Dan

1.hi dan there it's not possible to read a local list with the api but I can put in  a work around by making the input a string or something.

[rocket976 62]

How can I download the update Kev? Im already on your list , it wont let me subscribe again...

[Bot-Factory 602]

How can I download the update Kev? Im already on your list , it wont let me subscribe again...

You should have received an email.

 

Dan

[Bot-Factory 602]

Feature Request:

 

Clear all large lists (Similar to Clear all Large Tables)

 

Thanks

Dan

[kev123 132]

added to the list thanks

 

 

Feature Request:

 

Clear all large lists (Similar to Clear all Large Tables)

 

Thanks

Dan

 

[Bot-Factory 602]

Wow. The performance of the plugin is really outstanding!!

I have 76 million entries in a large list now. Plugin is using 7gb Memory now. 

 

And it's still running smoothly. 

 

Great job!

Dan

[kev123 132]

Wow. The performance of the plugin is really outstanding!!

I have 76 million entries in a large list now. Plugin is using 7gb Memory now. 

 

And it's still running smoothly. 

 

Great job!

Dan

wow

try sorting the list that's the ultimate test.   I would save what your doing thou if you have 76 million records could get messy.

[Bot-Factory 602]

wow

try sorting the list that's the ultimate test.   I would save what your doing thou if you have 76 million records could get messy.

Removing duplicates works nice as well.

 

I'm working on an universal scraping framework. using Http post to scrape data. And big table plugin to manage and save the data. 

I only enter the start url, the scrape regex and the nexturl regex. And hit run. 

 

Not sure how you handle that much data with great performance. When I think about the ubout table... Get's stuck after 20k entries already :-)

 

Dan

[Code Docta (Nick C.) 638]

Excellent work Kev123!!!!!!!!!!!!!!!!

 

 

I really am thankful for this plugin and appreciate all that you have done with it and the other plugins you do...

 

TC

[kev123 132]

 

 

I'm working on an universal scraping framework. using Http post to scrape data. And big table plugin to manage and save the data. 

I only enter the start url, the scrape regex and the nexturl regex. And hit run. 

 

 

I have this done for search engines reduced my coding massively. Add a new search engine takes a few min's. Other benefit is you can encrypt the "data file" and  save locally thus allowing silent updates.

[CTSchmidt 1]

I have a question: 

how to i add a specific row from LargeTableA to LargeTableB?

[Bot-Factory 602]

TIP:

 

Currently the command:

plugin command("Bigtable.dll", "ubot list to large list", "xx", %xx)

 

Only supports global ubot lists. So if you want to convert a local list into a large list you can use:

 

loop($list total(%tmp)) {
        plugin command("Bigtable.dll", "Add item to large list", "orderids", $next list item(%tmp))

    }

 

 

Kev already works on an update to address this.

[Bot-Factory 602]

I have a question: 

 

how to i add a specific row from LargeTableA to LargeTableB?

 

set(#loopcounter, 0, "Global")

loop($plugin function("Bigtable.dll", "Large Table Total Rows", "TableA")) {

    plugin command("Bigtable.dll", "Set Large Table cell", "TableB", #loopcounter, 0, $plugin function("Bigtable.dll", "Large Table Cell", "TableA", #loopcounter, 0))

    increment(#loopcounter)

}

 

Cheers

Dan

[UBotBuddy 331]

Hey Kev,

 

I am a little late to the party but I just subscribed.

 

I just blew out my UBot with a out of memory issue so I just happened to see this thread.  I am looking forward to testing the Large List as well as the Large Table functions.  Looks like it will save me a ton of coding work arounds.

 

Later!

 

Buddy

[kev123 132]

Hey Kev,

 

I am a little late to the party but I just subscribed.

 

I just blew out my UBot with a out of memory issue so I just happened to see this thread.  I am looking forward to testing the Large List as well as the Large Table functions.  Looks like it will save me a ton of coding work arounds.

 

Later!

 

Buddy

great the reason it was created was to deal with two things list sizes and working in threads with these items globally.

 

The next update (apart from a few additions ) is concentrated on using the tables and lists safely in threads at fast speeds this has come about from the smart thread plugin and the increased speed that threading is executed.

[CTSchmidt 1]

Sorry i did a mistake, I meant:

 

I have a LargeList with ID's (eg. 001, 002, 008)

 

now i want to add a row from LargeTableA (with a matching id) to LargeTableB

 

the result will be: LargeTableB: 

001,data,data
002,data,data
008,data,data

This was what I meant with "specific row"

 

Thanks in advance

 

CTSchmidt

Edited August 8, 2014 by CTSchmidt

[Bot-Factory 602]

Sorry i did a mistake, I meant:

 

I have a LargeList with ID's (eg. 001, 002, 008)

 

now i want to add a row from LargeTableA (with a matching id) to LargeTableB

 

the result will be: LargeTableB: 

001,data,data
002,data,data
008,data,data

This was what I meant with "specific row"

 

Thanks in advance

 

CTSchmidt

 

Well it's the same approach as the example I posted. You just have to apply the right logic to make it work. But the commands are the same.

Loop through the stuff you have. Read the data you need, and then add it at the position where you want it.

 

There is no other way to do that. 

 

Dan

[CTSchmidt 1]

okay I will try it, I will post my solution

 

btw great Stuff

 

CTSchmidt

[UBotBuddy 331]

While I rarely use Threads I love the fact that this reduces my overall memory footprint.  VERY cool Kev.

 

I did think of an enhancement if you decide to use it.  Actually two.

 

1. Provide a function like "Next Large List Item" like "$next list item".  That way we don't have to seed a variable in the "Large List Item".

 

2. Provide a way in suggestion #1 & your "Random Large List Item" to remove the item out of the List when it is selected by the command.

 

Hmmm

 

Just thought of a question.

 

How do I save a Large List to a file?

 

Thanks again!

 

Buddy

[Bot-Factory 602]

While I rarely use Threads I love the fact that this reduces my overall memory footprint.  VERY cool Kev.

 

I did think of an enhancement if you decide to use it.  Actually two.

 

1. Provide a function like "Next Large List Item" like "$next list item".  That way we don't have to seed a variable in the "Large List Item".

 

2. Provide a way in suggestion #1 & your "Random Large List Item" to remove the item out of the List when it is selected by the command.

 

Hmmm

 

Just thought of a question.

 

How do I save a Large List to a file?

 

Thanks again!

 

Buddy

 

save to file("d:\\x.csv", $plugin function("Bigtable.dll", "Large list return", "test"))

 

But I think kev works on a save large list to file command as well.

 

Dan

[UBotBuddy 331]

Thanks Dan!

[Bot-Factory 602]

HowTo Breakdown a Large List into smaller Chunks

 

plugin command("Bigtable.dll", "Clear large list", "testlist")

plugin command("Bigtable.dll", "Clear large list", "tmplist")
loop(20133) {
    plugin command("Bigtable.dll", "Add item to large list", "testlist", $random text(5))
}
alert($plugin function("Bigtable.dll", "Large list total", "testlist"))
set(#listcounter1, $plugin function("Bigtable.dll", "Large list total", "testlist"), "Global")
set(#listcounter2, 0, "Global")
loop while($comparison(#listcounter1, ">", 0)) {
    if($comparison(#listcounter1, ">=", 5000)) {
        then {
            loop(5000) {
                plugin command("Bigtable.dll", "Add item to large list", "tmplist", $plugin function("Bigtable.dll", "Large list item", "testlist", #listcounter2))
                increment(#listcounter2)
            }
            set(#listcounter1, $subtract(#listcounter1, 5000), "Global")
        }
        else {
            loop(#listcounter1) {
                plugin command("Bigtable.dll", "Add item to large list", "tmplist", $plugin function("Bigtable.dll", "Large list item", "testlist", #listcounter2))
                increment(#listcounter2)
            }
            set(#listcounter1, $subtract(#listcounter1, #listcounter1), "Global")
        }
    }
    alert($plugin function("Bigtable.dll", "Large list total", "tmplist"))
    plugin command("Bigtable.dll", "Clear large list", "tmplist")
}

 

 

 

You now could use the tmplist to launch your threads. This would allow you to scrape through a list of urls for example, but save the results in between. So if you initial list has 1 million entries, they are not executed in one go.

 

If you want to use this within threads, don't forget to change the Global variables to local ones!

 

If someone has an idea how to simplify / optimize that, please share.

 

Cheers

Dan

 

 

[kev123 132]

dan you could use my advanced file plugin and read in in chunks. if your reading from file

[Bot-Factory 602]

dan you could use my advanced file plugin and read in in chunks.

 

You mean saving the 2million urls to a tmp file. And then use "read part file" to load chunks of it?

Do you think that's faster than the loop thing?

 

Dan

[kev123 132]

You mean saving the 2million urls to a tmp file. And then use "read part file" to load chunks of it?

Do you think that's faster than the loop thing?

 

Dan

I edit my above comment before seeing your response. If your reading from file yes if this are already in memory no keep as is.