Jump to content
UBot Underground

[Sell] Smart Thread Plugin - Easy threading


Recommended Posts

  • Replies 184
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Hi James.   I converted the example from the thread you posted. I also removed some of the stuff that was in the script that is not directly required to get the job done:   So here's the ThreadPlugin

So in general. With multithreading there are a couple of things you should be aware of.   1. Threads are running at the same time. So you can't use global variables to store data.  2. Add data to a gl

Duplicate post. I replied in the ExBrowser Thread.   Dan

Awesome Plugin Kev. I had the luxury to test some of the beta versions so I wanted to share some details about it with you.

 

I already update some of my bots to use this plugin. So instead of doing all the thread management and if threadcount >= kind of stuff on your own, this plugin manages it all for you. 

 

So it's a lot easier and a lot faster to get your multithreading bots working. Specially for new users who haven't worked with threading a lot. 

But even for the advanced users this will reduce the complexity of your bots. Which is always a good thing to reduce errors. 

 

The advanced thread control gives you a lot of additional power. Because it's able to loop through lists and stuff automatically. So in the past you always needed some kind of loopcounter and things like next list item.You don't need that anymore now. 

 

And it also integrates nicely with the large list / large table plugin. Which I use in almost all my bots now because it reduces memory consumption by 90%.

 

So in my opinion this is a clear win.

 

Cheers

Dan

Link to post
Share on other sites

Just purchased cant wait to check it out bud.  NIce job.

Great hope you enjoy using it.

Awesome Plugin Kev. I had the luxury to test some of the beta versions so I wanted to share some details about it with you.

 

I already update some of my bots to use this plugin. So instead of doing all the thread management and if threadcount >= kind of stuff on your own, this plugin manages it all for you. 

 

So it's a lot easier and a lot faster to get your multithreading bots working. Specially for new users who haven't worked with threading a lot. 

But even for the advanced users this will reduce the complexity of your bots. Which is always a good thing to reduce errors. 

 

The advanced thread control give you a lot of additional power. Because it's able to loop through lists and stuff automatically. So in the past you always needed some kind of loopcounter and things like next list item.

You don't need that anymore now. 

 

And it also integrates nicely with the large list / large table plugin. Which I use in almost all my bots now because it reduces memory consumption by 90%.

 

Cheers

Dan

Thanks Dan, you where a great help in testing

Link to post
Share on other sites

I have a account creator id love to use this with,
Now the code I use generates a great deal of variables to make each account unique & save all the info to a xml file.
Would I be able to simply paste my code into the define & have it work multi threaded, or will i have to change a great deal of code?

Link to post
Share on other sites

I have a account creator id love to use this with,

Now the code I use generates a great deal of variables to make each account unique & save all the info to a xml file.

Would I be able to simply paste my code into the define & have it work multi threaded, or will i have to change a great deal of code?

it works like normal threads but a lot easier and faster. So you would still have to make it work with local variables etc. if you send me a example I can take a look and help out

Link to post
Share on other sites

I have a similar problem Black Automation.

 

I have a bot I'm trying to make it multi-threaded using the

plugin but so far I can't figure out how to make it work

correctly.

 

I can't do it with or without existing code so I'm a little

confused on how to use this plugin correctly I suppose.

 

But kev123 has been great in support and we'll tackle

this one again tomorrow when he's online.

  • Like 1
Link to post
Share on other sites

it works like normal threads but a lot easier and faster. So you would still have to make it work with local variables etc. if you send me a example I can take a look and help out

It would be great if you could share with us a sample script with local variables.

 

Thanks.

Link to post
Share on other sites

I have a similar problem Black Automation.

 

I have a bot I'm trying to make it multi-threaded using the

plugin but so far I can't figure out how to make it work

correctly.

 

I can't do it with or without existing code so I'm a little

confused on how to use this plugin correctly I suppose.

 

But kev123 has been great in support and we'll tackle

this one again tomorrow when he's online.

 

 

So in general. With multithreading there are a couple of things you should be aware of.

 

1. Threads are running at the same time. So you can't use global variables to store data. 

2. Add data to a global list (ubot list) from multiple threads at the same time could result in some data loss. 

3. Writing data to a file shouldn't be done from with the threads.

 

 

So here is the general process:

 

1. The stuff you scrape within your threads needs to be stored in a save way.

2. After all threads are finished you write the stuff to a file. 

 

I highly recommend that you take a look at the large data plugin. This plugin is thread safe. And can store millions of lines of data without slowing ubot down or anything.

 

 

Here's a quick example:

plugin command("Bigtable.dll""Clear all Large Tables")

plugin command("Bigtable.dll""Create Large Table""data", 50000, 2)

plugin command("Smartthreads.dll""Thread control normal""ThreadDefine", 10, 2, "No")

plugin command("Bigtable.dll""Large Table Save file""d:\\x.csv""data")

define ThreadDefine(#counter) {

    plugin command("Bigtable.dll""Set Large Table cell""data"#counter, 0, "Scrape Data 1")

    plugin command("Bigtable.dll""Set Large Table cell""data"#counter, 2, "Scrape Data 2")

}

 

 

 

Another example where you scrape URLs to a list first and then loop through that list(urls) to extract the data:

ScrapeDefine()

stop script()

define ScrapeDefine {

    plugin command("Bigtable.dll""Clear all Large Tables")

    plugin command("Bigtable.dll""Clear large list""tmp")

    plugin command("Smartthreads.dll""Thread control normal""ScrapeUrls", 10, 2, "No")

    plugin command("Smartthreads.dll""Thread control normal""ScrapeData"$plugin function("Bigtable.dll""Large list total""tmp"), 2, "No")

    plugin command("Bigtable.dll""Large Table Save file""d:\\x.csv""data")

}

define ScrapeUrls(#counter) {

    plugin command("Bigtable.dll""Add item to large list""tmp""use regex or xpath to scrape")

}

define ScrapeData(#counter) {

    plugin command("Bigtable.dll""Set Large Table cell""data"#counter, 0, "{$plugin function("Bigtable.dll""Large list item""tmp"#counter)}+use regex or xpath to scrape")

    plugin command("Bigtable.dll""Set Large Table cell""data"#counter, 2, "{$plugin function("Bigtable.dll""Large list item""tmp"#counter)}+use regex or xpath to scrape")

}

 

 

Hope that helps.

 

Dan

  • Like 2
Link to post
Share on other sites

Thank you Dan!

 

It means changing a LOT in my bot but hopefully it is worth it :)

 

You could also continue using the ubot tables if that's easier for you. But depending on how many data you scrape, the large list plugin will greatly enhance performance. 

But I'm not talking about bots who scrape 500 urls here :-)

 

This is more for bots who visit and scrape 100k+ Urls. 

 

Dan

Link to post
Share on other sites

You could also continue using the ubot tables if that's easier for you. But depending on how many data you scrape, the large list plugin will greatly enhance performance. 

But I'm not talking about bots who scrape 500 urls here :-)

 

This is more for bots who visit and scrape 100k+ Urls. 

 

Dan

 

If i want to search a data on a row using bigtables then how easy is it comparing if i'm using sqlite plugin to store and search a data Dan?

Link to post
Share on other sites

It would be great if you could share with us a sample script with local variables.

 

Thanks.

I'll create a separate thread when I get chance because this is relevant to ubot normal threads and the smart thread plugin.

 

I'll probably chuck in a google scraper as well for free in the source code of this plugin.

 

Are there any plans to integrate this with BotScriptFast?

Yes this and the large table plugin will be integrated with botscriptfast

 

This Multithreading code from TJ always works for me : http://www.ubotstudio.com/forum/index.php?/topic/10042-new-v4-tutorial-multi-threading-example/page-2&do=findComment&comment=73972

 

Now the question is how to convert it if using smartthread plugin Kev?

post a example in this thread and i'll do a conversion to show. check out the source code as well it shows so examples if you understand how to multithread using normal threads this will be very easy for you.

 

Ubot professional edition required for this plugin as per ubots rules

 

 

 

Link to post
Share on other sites

This Multithreading code from TJ always works for me : http://www.ubotstudio.com/forum/index.php?/topic/10042-new-v4-tutorial-multi-threading-example/page-2&do=findComment&comment=73972

 

Now the question is how to convert it if using smartthread plugin Kev?

 

Hi James.

 

I converted the example from the thread you posted.

I also removed some of the stuff that was in the script that is not directly required to get the job done:

 

So here's the ThreadPlugin Version. 

 

Create Lists()

plugin command("Smartthreads.dll""Thread control normal""codeprocedure"$list total(%List1), 5, "No")

define codeprocedure(#counter) {

    add list to list(%local break down$list from text($list item(%List1#counter), ","), "Don\'t Delete""Local")

    set(#VariableA$list item(%local break down, 0), "Local")

    set(#VariableB$list item(%local break down, 1), "Local")

    set(#VariableC$list item(%local break down, 2), "Local")

    add item to list(%recompile"{#VariableB},{#VariableC},{#VariableA}""Don\'t Delete""Global")

}

define Create Lists {

    clear list(%List1)

    clear list(%recompile)

    add list to list(%List1$list from text("A,26,z

B,25,y

C,24,x

D,23,w

E,22,v

F,21,u

G,20,t

H,19,s

I,18,r

J,17,q

K,16,p

L,15,o

M,14,n

N,13,m

O,12,l

P,11,k

Q,10,j

R,9,i

S,8,h

T,7,g

U,6,f

V,5,e

W,4,d

X,3,c

Y,2,b

Z,1,a"$new line), "Don\'t Delete""Global")

}

 

 

Let me know if there are any questions.

 

PS:  Writing to a ubot global list from within threads doesn't work 100% reliable all the time. I highly recommend to use large data plugin for this job.

 

Dan

  • Like 3
Link to post
Share on other sites

 

[FREE] Scrape Framework

 

Here is a free scraping framework for you guys. It's using

the smartthread plugin and the large data plugin from Kev.

 

DOWNLOAD

 

Great share dan i'll check it out. Pretty much if http related or processing large amounts of data dan knows his stuff

Link to post
Share on other sites

×
×
  • Create New...