kev123 132 Posted August 4, 2014 Report Share Posted August 4, 2014 http://robobest.com/wp-content/uploads/2014/08/Threeadplugintop.png http://robobest.com/wp-content/uploads/2014/08/threadpluginmiddle.png http://robobest.com/wp-content/uploads/2014/09/latestthreadbottom.png Ubot professional edition required for this plugin as per ubots rules 1 Quote Link to post Share on other sites
LoWrIdErTJ - BotGuru 904 Posted August 4, 2014 Report Share Posted August 4, 2014 Just purchased cant wait to check it out bud. NIce job. Quote Link to post Share on other sites
Bot-Factory 602 Posted August 4, 2014 Report Share Posted August 4, 2014 Awesome Plugin Kev. I had the luxury to test some of the beta versions so I wanted to share some details about it with you. I already update some of my bots to use this plugin. So instead of doing all the thread management and if threadcount >= kind of stuff on your own, this plugin manages it all for you. So it's a lot easier and a lot faster to get your multithreading bots working. Specially for new users who haven't worked with threading a lot. But even for the advanced users this will reduce the complexity of your bots. Which is always a good thing to reduce errors. The advanced thread control gives you a lot of additional power. Because it's able to loop through lists and stuff automatically. So in the past you always needed some kind of loopcounter and things like next list item.You don't need that anymore now. And it also integrates nicely with the large list / large table plugin. Which I use in almost all my bots now because it reduces memory consumption by 90%. So in my opinion this is a clear win. CheersDan Quote Link to post Share on other sites
kev123 132 Posted August 4, 2014 Author Report Share Posted August 4, 2014 Just purchased cant wait to check it out bud. NIce job.Great hope you enjoy using it.Awesome Plugin Kev. I had the luxury to test some of the beta versions so I wanted to share some details about it with you. I already update some of my bots to use this plugin. So instead of doing all the thread management and if threadcount >= kind of stuff on your own, this plugin manages it all for you. So it's a lot easier and a lot faster to get your multithreading bots working. Specially for new users who haven't worked with threading a lot. But even for the advanced users this will reduce the complexity of your bots. Which is always a good thing to reduce errors. The advanced thread control give you a lot of additional power. Because it's able to loop through lists and stuff automatically. So in the past you always needed some kind of loopcounter and things like next list item.You don't need that anymore now. And it also integrates nicely with the large list / large table plugin. Which I use in almost all my bots now because it reduces memory consumption by 90%. CheersDanThanks Dan, you where a great help in testing Quote Link to post Share on other sites
Kreatus (Ubot Ninja) 422 Posted August 4, 2014 Report Share Posted August 4, 2014 Nice one kev. Will purchase this one soon. Quote Link to post Share on other sites
Chris M 55 Posted August 4, 2014 Report Share Posted August 4, 2014 Just purchased! This is something I've been needing for a while now Thanks Kev! Quote Link to post Share on other sites
giganut 109 Posted August 4, 2014 Report Share Posted August 4, 2014 Looks great! I will come back and pic this up in a bit. Quote Link to post Share on other sites
ecbzh 47 Posted August 4, 2014 Report Share Posted August 4, 2014 Just purchased ! You have just made my coding life easier ... Quote Link to post Share on other sites
Bot-Factory 602 Posted August 4, 2014 Report Share Posted August 4, 2014 Don't forget to pickup his large data plugin as well!http://www.ubotstudio.com/forum/index.php?/topic/16308-free-plugin-large-data/ It's FREE!!! It will reduce memory consumption when working with large lists and tables by 95%! Dan 1 Quote Link to post Share on other sites
Code Docta (Nick C.) 638 Posted August 4, 2014 Report Share Posted August 4, 2014 Finally!!! my kind of threading...on my list of must haves.Should sell millions Kev good looking out man. Quote Link to post Share on other sites
Black Automation 20 Posted August 5, 2014 Report Share Posted August 5, 2014 I have a account creator id love to use this with,Now the code I use generates a great deal of variables to make each account unique & save all the info to a xml file.Would I be able to simply paste my code into the define & have it work multi threaded, or will i have to change a great deal of code? Quote Link to post Share on other sites
kev123 132 Posted August 5, 2014 Author Report Share Posted August 5, 2014 I have a account creator id love to use this with,Now the code I use generates a great deal of variables to make each account unique & save all the info to a xml file.Would I be able to simply paste my code into the define & have it work multi threaded, or will i have to change a great deal of code?it works like normal threads but a lot easier and faster. So you would still have to make it work with local variables etc. if you send me a example I can take a look and help out Quote Link to post Share on other sites
Bliss 194 Posted August 5, 2014 Report Share Posted August 5, 2014 Cant wait to find some time to try it.. Nice one Kev! Quote Link to post Share on other sites
Chris M 55 Posted August 5, 2014 Report Share Posted August 5, 2014 I have a similar problem Black Automation. I have a bot I'm trying to make it multi-threaded using theplugin but so far I can't figure out how to make it workcorrectly. I can't do it with or without existing code so I'm a littleconfused on how to use this plugin correctly I suppose. But kev123 has been great in support and we'll tacklethis one again tomorrow when he's online. 1 Quote Link to post Share on other sites
ecbzh 47 Posted August 5, 2014 Report Share Posted August 5, 2014 it works like normal threads but a lot easier and faster. So you would still have to make it work with local variables etc. if you send me a example I can take a look and help outIt would be great if you could share with us a sample script with local variables. Thanks. Quote Link to post Share on other sites
mrwalt 34 Posted August 5, 2014 Report Share Posted August 5, 2014 Are there any plans to integrate this with BotScriptFast? Quote Link to post Share on other sites
Bot-Factory 602 Posted August 5, 2014 Report Share Posted August 5, 2014 I have a similar problem Black Automation. I have a bot I'm trying to make it multi-threaded using theplugin but so far I can't figure out how to make it workcorrectly. I can't do it with or without existing code so I'm a littleconfused on how to use this plugin correctly I suppose. But kev123 has been great in support and we'll tacklethis one again tomorrow when he's online. So in general. With multithreading there are a couple of things you should be aware of. 1. Threads are running at the same time. So you can't use global variables to store data. 2. Add data to a global list (ubot list) from multiple threads at the same time could result in some data loss. 3. Writing data to a file shouldn't be done from with the threads. So here is the general process: 1. The stuff you scrape within your threads needs to be stored in a save way.2. After all threads are finished you write the stuff to a file. I highly recommend that you take a look at the large data plugin. This plugin is thread safe. And can store millions of lines of data without slowing ubot down or anything. Here's a quick example:plugin command("Bigtable.dll", "Clear all Large Tables")plugin command("Bigtable.dll", "Create Large Table", "data", 50000, 2)plugin command("Smartthreads.dll", "Thread control normal", "ThreadDefine", 10, 2, "No")plugin command("Bigtable.dll", "Large Table Save file", "d:\\x.csv", "data")define ThreadDefine(#counter) { plugin command("Bigtable.dll", "Set Large Table cell", "data", #counter, 0, "Scrape Data 1") plugin command("Bigtable.dll", "Set Large Table cell", "data", #counter, 2, "Scrape Data 2")} Another example where you scrape URLs to a list first and then loop through that list(urls) to extract the data:ScrapeDefine()stop script()define ScrapeDefine { plugin command("Bigtable.dll", "Clear all Large Tables") plugin command("Bigtable.dll", "Clear large list", "tmp") plugin command("Smartthreads.dll", "Thread control normal", "ScrapeUrls", 10, 2, "No") plugin command("Smartthreads.dll", "Thread control normal", "ScrapeData", $plugin function("Bigtable.dll", "Large list total", "tmp"), 2, "No") plugin command("Bigtable.dll", "Large Table Save file", "d:\\x.csv", "data")}define ScrapeUrls(#counter) { plugin command("Bigtable.dll", "Add item to large list", "tmp", "use regex or xpath to scrape")}define ScrapeData(#counter) { plugin command("Bigtable.dll", "Set Large Table cell", "data", #counter, 0, "{$plugin function("Bigtable.dll", "Large list item", "tmp", #counter)}+use regex or xpath to scrape") plugin command("Bigtable.dll", "Set Large Table cell", "data", #counter, 2, "{$plugin function("Bigtable.dll", "Large list item", "tmp", #counter)}+use regex or xpath to scrape")} Hope that helps. Dan 2 Quote Link to post Share on other sites
Chris M 55 Posted August 5, 2014 Report Share Posted August 5, 2014 Thank you Dan! It means changing a LOT in my bot but hopefully it is worth it Quote Link to post Share on other sites
Bot-Factory 602 Posted August 5, 2014 Report Share Posted August 5, 2014 Thank you Dan! It means changing a LOT in my bot but hopefully it is worth it You could also continue using the ubot tables if that's easier for you. But depending on how many data you scrape, the large list plugin will greatly enhance performance. But I'm not talking about bots who scrape 500 urls here :-) This is more for bots who visit and scrape 100k+ Urls. Dan Quote Link to post Share on other sites
jamesfar 15 Posted August 5, 2014 Report Share Posted August 5, 2014 This Multithreading code from TJ always works for me : http://www.ubotstudio.com/forum/index.php?/topic/10042-new-v4-tutorial-multi-threading-example/page-2&do=findComment&comment=73972 Now the question is how to convert it if using smartthread plugin Kev? Quote Link to post Share on other sites
jamesfar 15 Posted August 5, 2014 Report Share Posted August 5, 2014 You could also continue using the ubot tables if that's easier for you. But depending on how many data you scrape, the large list plugin will greatly enhance performance. But I'm not talking about bots who scrape 500 urls here :-) This is more for bots who visit and scrape 100k+ Urls. Dan If i want to search a data on a row using bigtables then how easy is it comparing if i'm using sqlite plugin to store and search a data Dan? Quote Link to post Share on other sites
kev123 132 Posted August 5, 2014 Author Report Share Posted August 5, 2014 It would be great if you could share with us a sample script with local variables. Thanks.I'll create a separate thread when I get chance because this is relevant to ubot normal threads and the smart thread plugin. I'll probably chuck in a google scraper as well for free in the source code of this plugin. Are there any plans to integrate this with BotScriptFast?Yes this and the large table plugin will be integrated with botscriptfast This Multithreading code from TJ always works for me : http://www.ubotstudio.com/forum/index.php?/topic/10042-new-v4-tutorial-multi-threading-example/page-2&do=findComment&comment=73972 Now the question is how to convert it if using smartthread plugin Kev?post a example in this thread and i'll do a conversion to show. check out the source code as well it shows so examples if you understand how to multithread using normal threads this will be very easy for you. Ubot professional edition required for this plugin as per ubots rules Quote Link to post Share on other sites
Bot-Factory 602 Posted August 5, 2014 Report Share Posted August 5, 2014 This Multithreading code from TJ always works for me : http://www.ubotstudio.com/forum/index.php?/topic/10042-new-v4-tutorial-multi-threading-example/page-2&do=findComment&comment=73972 Now the question is how to convert it if using smartthread plugin Kev? Hi James. I converted the example from the thread you posted.I also removed some of the stuff that was in the script that is not directly required to get the job done: So here's the ThreadPlugin Version. Create Lists()plugin command("Smartthreads.dll", "Thread control normal", "codeprocedure", $list total(%List1), 5, "No")define codeprocedure(#counter) { add list to list(%local break down, $list from text($list item(%List1, #counter), ","), "Don\'t Delete", "Local") set(#VariableA, $list item(%local break down, 0), "Local") set(#VariableB, $list item(%local break down, 1), "Local") set(#VariableC, $list item(%local break down, 2), "Local") add item to list(%recompile, "{#VariableB},{#VariableC},{#VariableA}", "Don\'t Delete", "Global")}define Create Lists { clear list(%List1) clear list(%recompile) add list to list(%List1, $list from text("A,26,zB,25,yC,24,xD,23,wE,22,vF,21,uG,20,tH,19,sI,18,rJ,17,qK,16,pL,15,oM,14,nN,13,mO,12,lP,11,kQ,10,jR,9,iS,8,hT,7,gU,6,fV,5,eW,4,dX,3,cY,2,bZ,1,a", $new line), "Don\'t Delete", "Global")} Let me know if there are any questions. PS: Writing to a ubot global list from within threads doesn't work 100% reliable all the time. I highly recommend to use large data plugin for this job. Dan 3 Quote Link to post Share on other sites
Bot-Factory 602 Posted August 5, 2014 Report Share Posted August 5, 2014 [FREE] Scrape Framework Here is a free scraping framework for you guys. It's usingthe smartthread plugin and the large data plugin from Kev. DOWNLOAD 1 Quote Link to post Share on other sites
kev123 132 Posted August 5, 2014 Author Report Share Posted August 5, 2014 [FREE] Scrape Framework Here is a free scraping framework for you guys. It's usingthe smartthread plugin and the large data plugin from Kev. DOWNLOAD Great share dan i'll check it out. Pretty much if http related or processing large amounts of data dan knows his stuff Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.