UBotDev 276 Posted January 24, 2014 Author Report Share Posted January 24, 2014 So when I write out from a thread it's local values to a global list/var/table cell - is that write operation - fully thread safe? Yes, it's safe. You only need to take care that you are not overwriting data. Quote Link to post Share on other sites
Sanjeev 46 Posted January 24, 2014 Report Share Posted January 24, 2014 Yes, it's safe. You only need to take care that you are not overwriting data. Therefore we pass a #row variable into the define before thread start - to track which table cell or list index to write back data into?This also ensures that data is written back in its proper order of processing back into the list/table? Quote Link to post Share on other sites
Sanjeev 46 Posted January 24, 2014 Report Share Posted January 24, 2014 THREAD START(#PROXIES ROW, #RESULTS ROW) increment(#PROXIES ROW) increment(#RESULTS ROW) decrement(#COUNT) would it be possible to use your 'threads counter plugin' to also decrement #count variable? Thanks. Quote Link to post Share on other sites
UBotDev 276 Posted January 24, 2014 Author Report Share Posted January 24, 2014 THREAD START(#PROXIES ROW, #RESULTS ROW) increment(#PROXIES ROW) increment(#RESULTS ROW) decrement(#COUNT) would it be possible to use your 'threads counter plugin' to also decrement #count variable? Thanks. Unfortunately no. Currently threads counter only provides 1 counter which you can use, like the name imposes. If you would also use it for count it would mess up your threads counter. Maybe this is something I can look into in the feature (currently quite busy so I can't promise anything yet), especially because UBot support doesn't think this is a bug in v5, so I don't think they will make any changes around that. Quote Link to post Share on other sites
Sanjeev 46 Posted January 24, 2014 Report Share Posted January 24, 2014 Maybe this is something I can look into in the feature (currently quite busy so I can't promise anything yet), Thanks UbotDev. Quote Link to post Share on other sites
mwickett 0 Posted March 7, 2014 Report Share Posted March 7, 2014 I'll echo thanks to UbotDev and other for making such great plugins and helping the community. Not sure that this is the place for it, but I'm using adapted example code from the start of this post (the second variant, I removed the wait comparison as indicated). It seems to run fine, but not too long into it, I get this: I saw this on a few other posts, but only in passing. I've tried it in a compiled bot and running right in uBot. Anyone else run into this? I am using CleanMem as suggested elsewhere to deal with out of memory errors. Running latest version of uBot 5 on Windows 8 (VM on OS X, loads of memory dedicated to the VM) Code is here: ui drop down("COUNT", "100,1000,10000,30000,2799", #Input Count) ui drop down("Threads", "10,20,30,40,50", #INPUT Threads) ui stat monitor("Threads (Active/Total):", "<b>{#THREADS Active}/{#THREADS Max}</b>") ui stat monitor("COUNT:", #COUNT) clear table(&RESULTS) add list to table as row(&RESULTS, 0, 0, $list from text("Count, Email, Name", ",")) set(#RESULTS ROW, 1, "Global") set(#COUNT, #Input Count, "Global") set(#THREADS Max, #INPUT Threads, "Global") comment("Reset counter to 0.") set(#THREADS Active, $plugin function("Threads Counter.dll", "threads counter", "reset"), "Global") comment("Main Threading Loop") loop while($comparison(#COUNT, ">", 0)) { if($comparison($plugin function("Threads Counter.dll", "threads counter", "read"), "<", #THREADS Max)) { then { comment("Increment Number Of Threads And Store Value to UBot Variable.") set(#THREADS Active, $plugin function("Threads Counter.dll", "threads counter", "increment"), "Global") thread_go(#COUNT, #RESULTS ROW) decrement(#COUNT) increment(#RESULTS ROW) } else { wait(0.2) } } } comment("Thread Command") define thread_go(#COUNT, #RESULTS ROW) { thread { set(#QueryURL, "URLGOESHERE={#COUNT}", "Local") plugin command("SocketCommands.dll", "socket container") { plugin command("SocketCommands.dll", "socket set header", "User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:27.0) Gecko/20100101 Firefox/27.0") plugin command("SocketCommands.dll", "socket set header", "Referer", "URL") plugin command("SocketCommands.dll", "socket navigate", "GET", #QueryURL) set(#HTML, $plugin function("SocketCommands.dll", "$socket page html"), "Local") if($contains(#HTML, "Record ID: {#COUNT}")) { then { set(#Email, $find regular expression($plugin function("SocketCommands.dll", "$socket page html"), "(?<=mailto:).*?(?=\")"), "Local") set(#Name, $find regular expression($plugin function("SocketCommands.dll", "$socket page html"), "(?<=Name</b></td>\\r\\n\\s\\s\\s\\s\\s\\s\\s\\s\\s\\s\\s\\s<td">).*?(?=</td>)"), "Local") set table cell(&RESULTS, #RESULTS ROW, 0, #COUNT) set table cell(&RESULTS, #RESULTS ROW, 1, #Email) set table cell(&RESULTS, #RESULTS ROW, 2, #Name) } else { set(#Email, "Failed", "Local") set(#Name, "Failed", "Local") set table cell(&RESULTS, #RESULTS ROW, 0, #COUNT) set table cell(&RESULTS, #RESULTS ROW, 1, #Email) set table cell(&RESULTS, #RESULTS ROW, 2, #Name) } } } comment("Decrement Number Of Threads And Store Value to UBot Variable.") set(#THREADS Active, $plugin function("Threads Counter.dll", "threads counter", "decrement"), "Global") } } comment("Wait For Threads To Close Or Timeout After 20 seconds.") set(#THREADS Active, $plugin function("Threads Counter.dll", "threads counter", "read"), "Global") loop while($comparison(#THREADS Active, ">", 0)) { set(#THREADS Active, $plugin function("Threads Counter.dll", "threads counter", "read"), "Global") wait(2) } comment("Check How Many Threads UBot Sees.") if($comparison(#THREADS Active, ">", 0)) { then { alert("Some threads were not closed/decremented properly.") } else { alert("All threads were closed") } } save to file("path", &RESULTS) stop script If anyone has any bright ideas, I'd sincerely appreciate it. Quote Link to post Share on other sites
UBotDev 276 Posted March 7, 2014 Author Report Share Posted March 7, 2014 I think I've already seen that error and I think it's a bug in UBot v5. If you run it in v4 it should work OK. 1 Quote Link to post Share on other sites
mwickett 0 Posted March 17, 2014 Report Share Posted March 17, 2014 Dumb question, I purchased v5 and so have never had v4 - can I get it somewhere? I submitted a support ticket, but thought I'd check here. Appreciate the help. Quote Link to post Share on other sites
UBotDev 276 Posted March 17, 2014 Author Report Share Posted March 17, 2014 Dumb question, I purchased v5 and so have never had v4 - can I get it somewhere? I submitted a support ticket, but thought I'd check here. Appreciate the help.I think they don't offer the downloads anymore, so contacting them via support ticket is the right way. Quote Link to post Share on other sites
mamica 10 Posted March 19, 2014 Report Share Posted March 19, 2014 I would like to know what was the maximum number of threads you run without problems. At high number like 100 or 200 threads, threading get corrupt. Bot hangs out at 200/200 but new browsers and threads are not visible in sidebar... (in other words it freeze) Any solution for this? Quote Link to post Share on other sites
jamesfar 15 Posted March 19, 2014 Report Share Posted March 19, 2014 I would like to know what was the maximum number of threads you run without problems. At high number like 100 or 200 threads, threading get corrupt. Bot hangs out at 200/200 but new browsers and threads are not visible in sidebar... (in other words it freeze) Any solution for this? you run 200 threads using new browser node? wow my max number on full browser based multithread is 10. above that mostly unstable 1 Quote Link to post Share on other sites
mamica 10 Posted March 19, 2014 Report Share Posted March 19, 2014 Yea i run it but it is not stable, it is not working at all. What if i run several bots with 10 threads? Quote Link to post Share on other sites
jamesfar 15 Posted March 19, 2014 Report Share Posted March 19, 2014 The problem with unstable thing is from ubot. Not from threadcounter plugin Quote Link to post Share on other sites
UBotDev 276 Posted March 19, 2014 Author Report Share Posted March 19, 2014 I would like to know what was the maximum number of threads you run without problems. At high number like 100 or 200 threads, threading get corrupt. Bot hangs out at 200/200 but new browsers and threads are not visible in sidebar... (in other words it freeze) Any solution for this?I've successfully ran 100 threads without any problems. When you run 200 threads, have you checked if your computer can handle it? Check RAM,CPU usage and if they are on 100% all the time the bot will become unresponsive... If that's the case you could try with a better computer... Quote Link to post Share on other sites
mamica 10 Posted March 20, 2014 Report Share Posted March 20, 2014 Yea, my comp (server) is very strong and when running bot i run at 30% with 100 threads. but i think jamesfar is saying, if you use in browser inside thread, then threads become unstable and we have problems and browser starts crashing and i think when browser crash, thread crash as well. So i can say that around 20 threads i dont see so much crashing. So maybe like 15 threads is OK. But to be honest that is kind of low. So any ideas how to increase that? Quote Link to post Share on other sites
jamesfar 15 Posted March 20, 2014 Report Share Posted March 20, 2014 Yea, my comp (server) is very strong and when running bot i run at 30% with 100 threads. but i think jamesfar is saying, if you use in browser inside thread, then threads become unstable and we have problems and browser starts crashing and i think when browser crash, thread crash as well. So i can say that around 20 threads i dont see so much crashing. So maybe like 15 threads is OK. But to be honest that is kind of low. So any ideas how to increase that? Perhaps runs on multiple computers or vps Quote Link to post Share on other sites
UBotDev 276 Posted March 20, 2014 Author Report Share Posted March 20, 2014 Yea, my comp (server) is very strong and when running bot i run at 30% with 100 threads. but i think jamesfar is saying, if you use in browser inside thread, then threads become unstable and we have problems and browser starts crashing and i think when browser crash, thread crash as well. So i can say that around 20 threads i dont see so much crashing. So maybe like 15 threads is OK. But to be honest that is kind of low. So any ideas how to increase that?Till today I only had one bot which was crashing constantly when running on 100 threads, and the solution was to close the bot after a while and re-open and restart it from where where it left off. This way I was able to process millions of inputs without any problem... 1 Quote Link to post Share on other sites
SaHiN 14 Posted April 24, 2014 Report Share Posted April 24, 2014 Very Good plugin. Thank you. 1 Question Aymen Http Post Plugin Http Page Scraper, $http get or $http post command proxy change don't stable. Problem Solve? help me? Thank you. Quote Link to post Share on other sites
UBotDev 276 Posted April 24, 2014 Author Report Share Posted April 24, 2014 Very Good plugin. Thank you. 1 Question Aymen Http Post Plugin Http Page Scraper, $http get or $http post command proxy change don't stable. Problem Solve? help me? Thank you.I don't think that's related to this plugin, since threads have nothing to do with HTTP POST plugin. So you should better ask Aymen if that plugin is not stable. Quote Link to post Share on other sites
SaHiN 14 Posted April 24, 2014 Report Share Posted April 24, 2014 I don't think that's related to this plugin, since threads have nothing to do with HTTP POST plugin. So you should better ask Aymen if that plugin is not stable. your plugin very Good dear, realy working. http://kilifcenter.com/proxy.png Problem = http post plugin Quote Link to post Share on other sites
earthlingj 13 Posted April 28, 2014 Report Share Posted April 28, 2014 This works really well when using lower threads, but when using my proxy checker @ 100+ threads, the count goes way off and ends up at 0 even though the process of checking proxies is far from complete. Quote Link to post Share on other sites
UBotDev 276 Posted April 28, 2014 Author Report Share Posted April 28, 2014 This works really well when using lower threads, but when using my proxy checker @ 100+ threads, the count goes way off and ends up at 0 even though the process of checking proxies is far from complete. Are you sure it's not your code? Paste it here is you want us to help you. I haven't had problems since using it, so I'm not sure what could be going wrong there... Quote Link to post Share on other sites
Bot-Factory 602 Posted April 28, 2014 Report Share Posted April 28, 2014 I used the plugin for one of my scraping bots with 100+ threads. And haven't had a single issue with it. If your "problem" is, that the active threads is lower than what you have configured as your target number, you could try to lower the wait time in this section: loop while($comparison($plugin function("Threads Counter.dll", "threads counter", "read"), ">=", 100)) { wait(0.5) } The more threads you run, the more impact this wait command has. So if that is set too high, it could result in an average of 94 threads instead of 100 threads. Bu this is not an issue. It's just that some threads are finishing while the wait command is still executing. Dan Quote Link to post Share on other sites
earthlingj 13 Posted April 28, 2014 Report Share Posted April 28, 2014 If i have the wait @ .5 then it skips some 'next list items' if i have the wait @ 1 then it doesnt skip next list items, but sometimes the used thread count goes in the negative?? Quote Link to post Share on other sites
Gogetta 263 Posted April 28, 2014 Report Share Posted April 28, 2014 I used the plugin for one of my scraping bots with 100+ threads. And haven't had a single issue with it. There is no problem with the threads counter plugin. But I am sure if you are using threads along with add to list there are problems with your scraping. The issue is even though the thread counter helps there is still a problem when adding data to a list at the exact same time as another thread. Run my code and you will see that not all of the items are being added to the list when running threads.Threads and Add Items (test).ubot Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.