Jump to content



Photo

[Sell] Smart Thread Plugin - Easy threading


  • Please log in to reply
182 replies to this topic

#1 kev123

kev123

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 739 posts
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v3.5 & v4.0
  • License:Developer Edition

Posted 04 August 2014 - 11:08 AM

Threeadplugintop.png

threadpluginmiddle.png

latestthreadbottom.png

 

Ubot professional edition required for this plugin as per ubots rules



#2 LoWrIdErTJ - BotGuru

LoWrIdErTJ - BotGuru

    Botguru.net

  • Moderators
  • 4679 posts
  • LocationMichigan
  • OS:Windows 10
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 04 August 2014 - 11:25 AM

Just purchased cant wait to check it out bud.  NIce job.


Web Automation Bots, and Ubot Plugins @ BotGuru.net Want a Custom bot? click here or Email me

ForumBanner.png

21 PLUGINS ABSOLUTELY FREE 100s for commands and functions
 


#3 Bot-Factory (Dan)

Bot-Factory (Dan)

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 3124 posts
  • LocationEurope/Germany
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v4.0
  • License:Developer Edition

Posted 04 August 2014 - 11:35 AM

Awesome Plugin Kev. I had the luxury to test some of the beta versions so I wanted to share some details about it with you.

 

I already update some of my bots to use this plugin. So instead of doing all the thread management and if threadcount >= kind of stuff on your own, this plugin manages it all for you. 

 

So it's a lot easier and a lot faster to get your multithreading bots working. Specially for new users who haven't worked with threading a lot. 

But even for the advanced users this will reduce the complexity of your bots. Which is always a good thing to reduce errors. 

 

The advanced thread control gives you a lot of additional power. Because it's able to loop through lists and stuff automatically. So in the past you always needed some kind of loopcounter and things like next list item.You don't need that anymore now. 

 

And it also integrates nicely with the large list / large table plugin. Which I use in almost all my bots now because it reduces memory consumption by 90%.

 

So in my opinion this is a clear win.

 

Cheers

Dan


Signature-JPG.jpg


#4 kev123

kev123

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 739 posts
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v3.5 & v4.0
  • License:Developer Edition

Posted 04 August 2014 - 11:41 AM

Just purchased cant wait to check it out bud.  NIce job.

Great hope you enjoy using it.

Awesome Plugin Kev. I had the luxury to test some of the beta versions so I wanted to share some details about it with you.

 

I already update some of my bots to use this plugin. So instead of doing all the thread management and if threadcount >= kind of stuff on your own, this plugin manages it all for you. 

 

So it's a lot easier and a lot faster to get your multithreading bots working. Specially for new users who haven't worked with threading a lot. 

But even for the advanced users this will reduce the complexity of your bots. Which is always a good thing to reduce errors. 

 

The advanced thread control give you a lot of additional power. Because it's able to loop through lists and stuff automatically. So in the past you always needed some kind of loopcounter and things like next list item.

You don't need that anymore now. 

 

And it also integrates nicely with the large list / large table plugin. Which I use in almost all my bots now because it reduces memory consumption by 90%.

 

Cheers

Dan

Thanks Dan, you where a great help in testing



#5 Kreatus (Ubot Ninja)

Kreatus (Ubot Ninja)

    Custombotsolutions.com

  • Fellow UBotter
  • PipPipPip
  • 3193 posts
  • OS:Windows 10
  • Total Memory:More Than 9Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 04 August 2014 - 12:19 PM

Nice one kev. Will purchase this one soon.



#6 Chris M

Chris M

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 497 posts
  • OS:Windows 10
  • Total Memory:More Than 9Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 04 August 2014 - 12:37 PM

Just purchased! This is something I've been needing for a while now :)

 

Thanks Kev!



#7 giganut

giganut

    softwareautomation.org

  • Fellow UBotter
  • PipPipPip
  • 535 posts
  • LocationLost In Space!
  • OS:Windows 10
  • Total Memory:4Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 04 August 2014 - 01:14 PM

Looks great! I will come back and pic this up in a bit. :)


softwareautomation-banner.png


#8 ecbzh

ecbzh

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 137 posts
  • LocationFrance
  • OS:Windows 7
  • Total Memory:3Gb
  • Framework:v3.5 & v4.0
  • License:Developer Edition

Posted 04 August 2014 - 01:48 PM

Just purchased ! You have just made my coding life easier ...

#9 Bot-Factory (Dan)

Bot-Factory (Dan)

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 3124 posts
  • LocationEurope/Germany
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v4.0
  • License:Developer Edition

Posted 04 August 2014 - 01:52 PM

Don't forget to pickup his large data plugin as well!

http://www.ubotstudi...gin-large-data/

 

It's FREE!!!

 

It will reduce memory consumption when working with large lists and tables by 95%!

 

Dan


Signature-JPG.jpg


#10 Code Docta (Nick C.)

Code Docta (Nick C.)

    Member for Hire

  • Fellow UBotter
  • PipPipPip
  • 1489 posts
  • LocationIn a Python or UBot IDE or ubotdocta.com
  • OS:Windows 10
  • Total Memory:More Than 9Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 04 August 2014 - 02:25 PM

Finally!!! my kind of threading...on my list of must haves.

Should sell millions Kev good looking out man.


UBot Expert Tutorials Store

 

ubot-docta-header-ubot-expert-store.png

 

Python and UBot expert tutorials and documentation


#11 Black Automation

Black Automation

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 192 posts
  • LocationUSA
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v4.0
  • License:Developer Edition

Posted 04 August 2014 - 05:06 PM

I have a account creator id love to use this with,
Now the code I use generates a great deal of variables to make each account unique & save all the info to a xml file.
Would I be able to simply paste my code into the define & have it work multi threaded, or will i have to change a great deal of code?



#12 kev123

kev123

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 739 posts
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v3.5 & v4.0
  • License:Developer Edition

Posted 04 August 2014 - 06:20 PM

I have a account creator id love to use this with,
Now the code I use generates a great deal of variables to make each account unique & save all the info to a xml file.
Would I be able to simply paste my code into the define & have it work multi threaded, or will i have to change a great deal of code?

it works like normal threads but a lot easier and faster. So you would still have to make it work with local variables etc. if you send me a example I can take a look and help out



#13 Bliss

Bliss

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 590 posts
  • LocationUK
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v3.5 & v4.0
  • License:Developer Edition

Posted 04 August 2014 - 07:52 PM

Cant wait to find some time to try it..

 

Nice one Kev!


WPMasterControl-45Off.png


#14 Chris M

Chris M

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 497 posts
  • OS:Windows 10
  • Total Memory:More Than 9Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 04 August 2014 - 09:13 PM

I have a similar problem Black Automation.

 

I have a bot I'm trying to make it multi-threaded using the

plugin but so far I can't figure out how to make it work

correctly.

 

I can't do it with or without existing code so I'm a little

confused on how to use this plugin correctly I suppose.

 

But kev123 has been great in support and we'll tackle

this one again tomorrow when he's online.



#15 ecbzh

ecbzh

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 137 posts
  • LocationFrance
  • OS:Windows 7
  • Total Memory:3Gb
  • Framework:v3.5 & v4.0
  • License:Developer Edition

Posted 04 August 2014 - 10:42 PM

it works like normal threads but a lot easier and faster. So you would still have to make it work with local variables etc. if you send me a example I can take a look and help out


It would be great if you could share with us a sample script with local variables.

Thanks.

#16 mrwalt

mrwalt

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 190 posts
  • OS:Windows 10
  • Total Memory:More Than 9Gb
  • Framework:v4.0
  • License:Developer Edition

Posted 04 August 2014 - 11:11 PM

Are there any plans to integrate this with BotScriptFast?



#17 Bot-Factory (Dan)

Bot-Factory (Dan)

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 3124 posts
  • LocationEurope/Germany
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v4.0
  • License:Developer Edition

Posted 05 August 2014 - 12:13 AM

I have a similar problem Black Automation.

 

I have a bot I'm trying to make it multi-threaded using the

plugin but so far I can't figure out how to make it work

correctly.

 

I can't do it with or without existing code so I'm a little

confused on how to use this plugin correctly I suppose.

 

But kev123 has been great in support and we'll tackle

this one again tomorrow when he's online.

 

 

So in general. With multithreading there are a couple of things you should be aware of.

 

1. Threads are running at the same time. So you can't use global variables to store data. 

2. Add data to a global list (ubot list) from multiple threads at the same time could result in some data loss. 

3. Writing data to a file shouldn't be done from with the threads.

 

 

So here is the general process:

 

1. The stuff you scrape within your threads needs to be stored in a save way.

2. After all threads are finished you write the stuff to a file. 

 

I highly recommend that you take a look at the large data plugin. This plugin is thread safe. And can store millions of lines of data without slowing ubot down or anything.

 

 

Here's a quick example:

plugin command("Bigtable.dll""Clear all Large Tables")
plugin command("Bigtable.dll""Create Large Table""data", 50000, 2)
plugin command("Smartthreads.dll""Thread control normal""ThreadDefine", 10, 2, "No")
plugin command("Bigtable.dll""Large Table Save file""d:\\x.csv""data")
define ThreadDefine(#counter) {
    plugin command("Bigtable.dll""Set Large Table cell""data"#counter, 0, "Scrape Data 1")
    plugin command("Bigtable.dll""Set Large Table cell""data"#counter, 2, "Scrape Data 2")
}

 

 

 

Another example where you scrape URLs to a list first and then loop through that list(urls) to extract the data:

ScrapeDefine()
stop script()
define ScrapeDefine {
    plugin command("Bigtable.dll""Clear all Large Tables")
    plugin command("Bigtable.dll""Clear large list""tmp")
    plugin command("Smartthreads.dll""Thread control normal""ScrapeUrls", 10, 2, "No")
    plugin command("Smartthreads.dll""Thread control normal""ScrapeData"$plugin function("Bigtable.dll""Large list total""tmp"), 2, "No")
    plugin command("Bigtable.dll""Large Table Save file""d:\\x.csv""data")
}
define ScrapeUrls(#counter) {
    plugin command("Bigtable.dll""Add item to large list""tmp""use regex or xpath to scrape")
}
define ScrapeData(#counter) {
    plugin command("Bigtable.dll""Set Large Table cell""data"#counter, 0, "{$plugin function("Bigtable.dll""Large list item""tmp"#counter)}+use regex or xpath to scrape")
    plugin command("Bigtable.dll""Set Large Table cell""data"#counter, 2, "{$plugin function("Bigtable.dll""Large list item""tmp"#counter)}+use regex or xpath to scrape")
}

 

 

Hope that helps.

 

Dan


Signature-JPG.jpg


#18 Chris M

Chris M

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 497 posts
  • OS:Windows 10
  • Total Memory:More Than 9Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 05 August 2014 - 01:43 AM

Thank you Dan!

 

It means changing a LOT in my bot but hopefully it is worth it :)



#19 Bot-Factory (Dan)

Bot-Factory (Dan)

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 3124 posts
  • LocationEurope/Germany
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v4.0
  • License:Developer Edition

Posted 05 August 2014 - 02:10 AM

Thank you Dan!

 

It means changing a LOT in my bot but hopefully it is worth it :)

 

You could also continue using the ubot tables if that's easier for you. But depending on how many data you scrape, the large list plugin will greatly enhance performance. 

But I'm not talking about bots who scrape 500 urls here :-)

 

This is more for bots who visit and scrape 100k+ Urls. 

 

Dan


Signature-JPG.jpg


#20 jamesfar

jamesfar

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 527 posts
  • OS:Windows 8
  • Total Memory:More Than 9Gb
  • Framework:v4.0
  • License:Developer Edition

Posted 05 August 2014 - 02:12 AM

This Multithreading code from TJ always works for me : http://www.ubotstudi...ge-2#entry73972

 

Now the question is how to convert it if using smartthread plugin Kev?


The best proxies provider so far.
* Fast and Great Support. * No software restriction.

* It is favourite proxies provider on GSA forum. * Almost 2000 positive reviews on fiverr. * More than 1000s active clients

 

Try It !





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users