Jump to content
UBot Underground

[Free] Plugin Large Data


Recommended Posts

Since there are list functions within this plugin what is the best way to use them as I have been trying for awhile 

and still can not figure it out as the list functions within this plugin do not work like the regular one's already built 

into Ubot. 

 

I know that most people use the table functions but I'm curious since they exist how do you use the list functions.

 

Does anyone have any good examples as in the plugin there were examples for the tables but none 

for the list?

Link to post
Share on other sites
  • Replies 209
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

My latest free plugin good for large table data storing   commands and functions are   http://robobest.com/wp-content/uploads/2014/04/bigdatacommands.png   http://robobest.com/wp-content/uploads/2014/

I edit my above comment before seeing your response. If your reading from file yes if this are already in memory no keep as is.

Much Appreciated Work KEV downloaded and will try now   

Since there are list functions within this plugin what is the best way to use them as I have been trying for awhile 

and still can not figure it out as the list functions within this plugin do not work like the regular one's already built 

into Ubot. 

 

I know that most people use the table functions but I'm curious since they exist how do you use the list functions.

 

Does anyone have any good examples as in the plugin there were examples for the tables but none 

for the list?

 

It would be helpful if you could explain where you struggle exactly? The list function is pretty similar to the one in ubot studio.

The only thing that is not there is Next List item. So you just have to use a counter instead.

 

Here's a basic example:

 

plugin command("Bigtable.dll""Clear all large list")

loop(20) {

    plugin command("Bigtable.dll""Add item to large list""test"$random text(5))

}

alert($plugin function("Bigtable.dll""Large list total""test"))

set(#counter, 0, "Global")

loop($plugin function("Bigtable.dll""Large list total""test")) {

    load html($plugin function("Bigtable.dll""Large list item""test"#counter))

    increment(#counter)

    wait(0.2)

}

 

 

Cheers

Dan

Link to post
Share on other sites
Is there a faster way to create a bigger Table?


 



plugin command("Bigtable.dll", "Large Table From file", #openFile, "largeDataSmall")
plugin command("Bigtable.dll", "Create Large Table", "largeData", $plugin function("Bigtable.dll", "Large Table Total Rows", "largeDataSmall"), 88)
set(#countList, 0, "Global")
loop($plugin function("Bigtable.dll", "Large Table Total Rows", "largeDataSmall")) {
set(#countInner, 0, "Global")
loop($plugin function("Bigtable.dll", "Large Table Total Columns", "largeDataSmall")) {
plugin command("Bigtable.dll", "Set Large Table cell", "largeData", #countList, #countInner, $plugin function("Bigtable.dll", "Large Table Cell", "largeDataSmall", #countList, #countInner))
increment(#countInner)
}
increment(#countList)
}



Thanks in advance!

Edited by CTSchmidt
Link to post
Share on other sites

 

Is there a faster way to create a bigger Table?
 
plugin command("Bigtable.dll", "Large Table From file", #openFile, "largeDataSmall")
plugin command("Bigtable.dll", "Create Large Table", "largeData", $plugin function("Bigtable.dll", "Large Table Total Rows", "largeDataSmall"), 88)
set(#countList, 0, "Global")
loop($plugin function("Bigtable.dll", "Large Table Total Rows", "largeDataSmall")) {
    set(#countInner, 0, "Global")
    loop($plugin function("Bigtable.dll", "Large Table Total Columns", "largeDataSmall")) {
        plugin command("Bigtable.dll", "Set Large Table cell", "largeData", #countList, #countInner, $plugin function("Bigtable.dll", "Large Table Cell", "largeDataSmall", #countList, #countInner))
        increment(#countInner)
    }
    increment(#countList)
}
Thanks in advance!

 

 

Hi.

 

If you want to add one table to another:

plugin command("Bigtable.dll""Large Table From file"#openFile"largeDataSmall")

plugin command("Bigtable.dll""Create Large Table""largeData"$plugin function("Bigtable.dll""Large Table Total Rows""largeDataSmall"), $plugin function("Bigtable.dll""Large Table Total Columns""largeDataSmall"))

plugin command("Bigtable.dll""Combine large tables""largeData""largeDataSmall")

 

 

Not sure if that's what you try to archive? But that basically is similar to your code. Just faster :-)

 

Dan

Link to post
Share on other sites

Hi.

 

If you want to add one table to another:

plugin command("Bigtable.dll""Large Table From file"#openFile"largeDataSmall")

plugin command("Bigtable.dll""Create Large Table""largeData"$plugin function("Bigtable.dll""Large Table Total Rows""largeDataSmall"), $plugin function("Bigtable.dll""Large Table Total Columns""largeDataSmall"))

plugin command("Bigtable.dll""Combine large tables""largeData""largeDataSmall")

 

 

Not sure if that's what you try to archive? But that basically is similar to your code. Just faster :-)

 

Dan

 

A okay, now I understand the function! If the tables are combined, the combined table is as large as both together. I just added some more columns.

 

Thanks Dan!

 

CTSchmidt

Link to post
Share on other sites
  • 2 weeks later...

Hi Kev, thanks for the plugin.

 

With ubots default tables we can insert a new row into a specific position. Is there a way to do this with a large table?

If not any plans to add it?

 

Cheers.

not currently (you can for lists) but you can use "set table cell" and loop through the row as a current alternative

Link to post
Share on other sites
  • 1 month later...

Hi Kev,

 

Is it safe to use large lists and large tables in threads? I'm just about to convert a single threaded bot to multi-threaded and thought I'd ask before hitting any stumbling locks.

 

While I'm at it... Do large lists / large tables work okay inside threads using your Smart Threads plugin?

Link to post
Share on other sites

yes thread safe, a lot of people are using it in multi thread bots. if you want to make tables/lists local to threads you will need to create you own unique identifier. Probably not needed depending on what you are doing.

  • Like 1
Link to post
Share on other sites
  • 3 weeks later...

I'm having a problem which my last thought could be the plugin.

 

Problem:

I have a var with list items after being scrapped. it looks like the following: 

​Username #1

 

​Username #1

 

​Username #2

 

​Username #2

 

Etc.

 

I add Large List from Text and then remove the duplicates. Note, this is multithreaded, 1-90 threads adding to the list. The Duplicate function is after the threads have finished. 

Anyways, the list is always off. Same list in the same spot. I thought the thread was running slower then the others, but the same thing happens. 

Example:
KEY: 
#1 = Thread 1
#2 = Thread 2
#3 = Thread 3

User = The User List For That Thread

#1 starts up
#2 starts up seconds after
#3 starts up seconds after

 

#1 scrapes user data and adds it to the list 
#2 scrapes user data and adds it to the list
#3 scrapes user data and adds it to the list 

List formate should be: 

#1 user
#2 user
#3 user 

But the end list,

#2 user
#3 user
#1 user

Same order for all 3 threads. If I go up to 4, would be #2 #3 #4 #1 

I'm trying to make an example that's not my current code. 

Regards,
HaHaItsJake

Link to post
Share on other sites

I'm having a problem which my last thought could be the plugin.

 

Problem:

I have a var with list items after being scrapped. it looks like the following: 

 

​Username #1

 

​Username #1

 

​Username #2

 

​Username #2

 

Etc.

 

I add Large List from Text and then remove the duplicates. Note, this is multithreaded, 1-90 threads adding to the list. The Duplicate function is after the threads have finished. 

 

Anyways, the list is always off. Same list in the same spot. I thought the thread was running slower then the others, but the same thing happens. 

 

Example:

KEY: 

#1 = Thread 1

#2 = Thread 2

#3 = Thread 3

User = The User List For That Thread

 

#1 starts up

#2 starts up seconds after

#3 starts up seconds after

 

#1 scrapes user data and adds it to the list 

#2 scrapes user data and adds it to the list

#3 scrapes user data and adds it to the list 

 

List formate should be: 

 

#1 user

#2 user

#3 user 

 

But the end list,

 

#2 user

#3 user

#1 user

 

Same order for all 3 threads. If I go up to 4, would be #2 #3 #4 #1 

 

I'm trying to make an example that's not my current code. 

 

Regards,

HaHaItsJake

 

 

Hi.

 

You never know when a thread finishes and how long something exactly takes. If you want to make that reliable you have to work with a counter. 

And use a large table. 

 

So if you run 5 threads your counter starts with 0 and goes up to 4.

And then within your thread you do:

Thread 1 = Writes to table row #counter (0), column 0

Thread 2 = Writes to table row #counter (1), column 0

Thread 3 = Writes to table row #counter (2), column 0

Thread 4 = Writes to table row #counter (3), column 0

Thread 5 = Writes to table row #counter (4), column 0

 

 

That way you can sort everything how you like it.

 

Another thing I wanted to mention, not all large data commands are thread save. 

So I would not use large list from text within a thread to save data to a shared large data list.

 

I normally scrape my data to a local large list (work with a counter for the list name) and then I use add large list to large list at the end of the thread. 

 

Dan

Link to post
Share on other sites

Hi.

 

You never know when a thread finishes and how long something exactly takes. If you want to make that reliable you have to work with a counter. 

And use a large table. 

 

So if you run 5 threads your counter starts with 0 and goes up to 4.

And then within your thread you do:

Thread 1 = Writes to table row #counter (0), column 0

Thread 2 = Writes to table row #counter (1), column 0

Thread 3 = Writes to table row #counter (2), column 0

Thread 4 = Writes to table row #counter (3), column 0

Thread 5 = Writes to table row #counter (4), column 0

 

 

That way you can sort everything how you like it.

 

Another thing I wanted to mention, not all large data commands are thread save. 

So I would not use large list from text within a thread to save data to a shared large data list.

 

I normally scrape my data to a local large list (work with a counter for the list name) and then I use add large list to large list at the end of the thread. 

 

Dan

That was my first though, which I still think it might be the case. Just all the testing to see if it's the thread speed came up with different data. Also might be along side the command used which not thread safe.  

 

How could you set/get thread "ID while using the Smart Threading by Kev. I have a way, but it's 3 define complicated. Is there a more easier way to get/set it? 

 

Regards,

HaHaItsJake

Link to post
Share on other sites

Smart Thread plugin always sends a counter to the thread. You can combine that into the large data table/list name.

 

Example:

define Define1(#counter) {
    plugin command("Bigtable.dll""Add item to large list""local1{#counter}""xx")
    plugin command("Bigtable.dll""Large list Remove duplicates""local1{#counter}")
    plugin command("Bigtable.dll""Add Large list to list""mainlist""local1{#counter}")
}
define Define2(#counter) {
    plugin command("Bigtable.dll""Add item to large list""local2{#counter}""xx")
    plugin command("Bigtable.dll""Large list Remove duplicates""local2{#counter}")
    plugin command("Bigtable.dll""Add Large list to list""mainlist""local2{#counter}")
}
define Define3(#counter) {
    plugin command("Bigtable.dll""Add item to large list""local3{#counter}""xx")
    plugin command("Bigtable.dll""Large list Remove duplicates""local2{#counter}")
    plugin command("Bigtable.dll""Add Large list to list""mainlist""local2{#counter}")
}

 

Cheers

Dan

Link to post
Share on other sites

...

 

Another thing I wanted to mention, not all large data commands are thread save. 

So I would not use large list from text within a thread to save data to a shared large data list.

 

Dan / Kev,

 

Can you mention which commands are thread safe and which are not?

  • Like 1
Link to post
Share on other sites

Dan / Kev,

 

Can you mention which commands are thread safe and which are not?

 

I'm not 100% sure, but I think Kev told me, that 

large list from (text,regex,xpath..)

large table from .....

 

shouldn't be used to write to the same list/table from within threads.

So I use those commands to write to a local list/table within my threads and then use 

add large list to list

combine large tables

 

To merge that stuff.

 

But I think Kev can give us the full picture here.

 

Dan

  • Like 1
Link to post
Share on other sites

anything that combines the tables/lists together and the add item to list command.(these lock at a command/function level) 

 

obviously stuff like setting table cells and reading aren't set so one single thread can only write at a time because this would cause them to slow but how there used, they don't need to be made thread safe.(you would never write and read to the same cell at the same time).

 

there is also a lower level of control/locking in the plugin to stop errors when add removing new tables/lists this is more to ensure the plugin stays effective at very high thread counts with a lot of tables/lists. something that won't be noticeable but thought I would mention.

 

thanks

kev

Link to post
Share on other sites

Hi Kev,

 

Ubot v5.5.7 just updated on my machine and it seems to be causing issues with the large data plugin.

 

When running code using the run button I get issues with large tables and large lists. See this example... http://screencast.com/t/oPg3mWz0Yl

 

When I run each command separately they work. If I run the code again after manually execution some times it works some time not.

 

I don't know if this is a v5.5.7 issue, a plugin issue or just my machine!

 

Anyone else reporting the same?

Edited by Pete_UK
Link to post
Share on other sites

Hi Kev,

 

Ubot v5.5.7 just updated on my machine and it seems to be causing issues with the large data plugin.

 

When running code using the run button I get issues with large tables and large lists. See this example... http://screencast.com/t/oPg3mWz0Yl

 

When I run each command separately they work. If I run the code again after manually execution some times it works some time not.

 

I don't know if this is a v5.5.7 issue, a plugin issue or just my machine!

 

Anyone else reporting the same?

 

Can you prepare a Ubot File where this can be reproduced? I haven't seen any issues with 5.5.7 so far. And I'm currently working on a new project where I use large data plugin a lot.

Dan

Link to post
Share on other sites

I can confirm this issue buts not specific to this plugin. Happened when I open a project that wasn't created in latest version of ubot. To solve I switched from code view and then back to node view cleared other similar issues I was having with other plugins.

Link to post
Share on other sites

I can confirm this issue buts not specific to this plugin. Happened when I open a project that wasn't created in latest version of ubot. To solve I switched from code view and then back to node view cleared other similar issues I was having with other plugins.

 

I think you are right Kev. When I tried to make a new bot... it all worked.

 

Had to switch between code and node view on all 7 tabs several times to get my broken bot working again.

 

Panic over!

 

Cheers.

Link to post
Share on other sites
  • 1 month later...

hi kev,

is there a way to get a list of any large tables or list that have been created.

if the bot stops or anything like that there is no way to see if a large table exist to check if it needs to be created or not :D

Link to post
Share on other sites

the latest version has check to see if list or table exists. working well in one of my bots. i'll probably send it out next week when I get chance to add one more feature.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...