Jump to content
UBot Underground

[FREE PLUGIN] Threads Counter - UBot v4 Threading Fixed


Recommended Posts

So when I write out from a thread it's local values to  a global list/var/table cell - is that write operation - fully  thread safe?

 

Yes, it's safe. You only need to take care that you are not overwriting data.

Link to post
Share on other sites
  • Replies 160
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

As you may be aware threading in UBot v4 is not safe and may not work as expect under certain conditions, which would result in threads slowly dying out, so at the end, you only get left with 1 active

Awesome plugin UBotDev

You are not using the plugin from the title, instead you are using the plugin that Blumi 40 keeps posting here (Blumi40, can you please open another thread for that plugin instead of discussing it her

 

Yes, it's safe. You only need to take care that you are not overwriting data.

 

Therefore we pass a #row variable into the define before thread start - to track which table cell or list index to write back data into?

This also ensures that data is written back in its proper order of processing back into the list/table?

Link to post
Share on other sites
THREAD START(#PROXIES ROW, #RESULTS ROW)
            increment(#PROXIES ROW)
            increment(#RESULTS ROW)
            decrement(#COUNT)

would it be possible to use your 'threads counter plugin'  to also decrement #count variable?

 

Thanks.

Link to post
Share on other sites
THREAD START(#PROXIES ROW, #RESULTS ROW)
            increment(#PROXIES ROW)
            increment(#RESULTS ROW)
            decrement(#COUNT)

would it be possible to use your 'threads counter plugin'  to also decrement #count variable?

 

Thanks.

 

Unfortunately no. Currently threads counter only provides 1 counter which you can use, like the name imposes. If you would also use it for count it would mess up your threads counter.

 

Maybe this is something I can look into in the feature (currently quite busy so I can't promise anything yet), especially because UBot support doesn't think this is a bug in v5, so I don't think they will make any changes around that.

Link to post
Share on other sites
  • 1 month later...

I'll echo thanks to UbotDev and other for making such great plugins and helping the community.

 

Not sure that this is the place for it, but I'm using adapted example code from the start of this post (the second variant, I removed the wait comparison as indicated). It seems to run fine, but not too long into it, I get this:

 

error.png

 

I saw this on a few other posts, but only in passing. I've tried it in a compiled bot and running right in uBot. Anyone else run into this?

 

I am using CleanMem as suggested elsewhere to deal with out of memory errors. Running latest version of uBot 5 on Windows 8 (VM on OS X, loads of memory dedicated to the VM)

 

Code is here:

ui drop down("COUNT", "100,1000,10000,30000,2799", #Input Count)
ui drop down("Threads", "10,20,30,40,50", #INPUT Threads)
ui stat monitor("Threads (Active/Total):", "<b>{#THREADS Active}/{#THREADS Max}</b>")
ui stat monitor("COUNT:", #COUNT)
clear table(&RESULTS)
add list to table as row(&RESULTS, 0, 0, $list from text("Count, Email, Name", ","))
set(#RESULTS ROW, 1, "Global")
set(#COUNT, #Input Count, "Global")
set(#THREADS Max, #INPUT Threads, "Global")
comment("Reset counter to 0.")
set(#THREADS Active, $plugin function("Threads Counter.dll", "threads counter", "reset"), "Global")
comment("Main Threading Loop")
loop while($comparison(#COUNT, ">", 0)) {
    if($comparison($plugin function("Threads Counter.dll", "threads counter", "read"), "<", #THREADS Max)) {
        then {
            comment("Increment Number Of Threads And Store Value to UBot Variable.")
            set(#THREADS Active, $plugin function("Threads Counter.dll", "threads counter", "increment"), "Global")
            thread_go(#COUNT, #RESULTS ROW)
            decrement(#COUNT)
            increment(#RESULTS ROW)
        }
        else {
            wait(0.2)
        }
    }
}
comment("Thread Command")
define thread_go(#COUNT, #RESULTS ROW) {
    thread {
        set(#QueryURL, "URLGOESHERE={#COUNT}", "Local")
        plugin command("SocketCommands.dll", "socket container") {
            plugin command("SocketCommands.dll", "socket set header", "User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:27.0) Gecko/20100101 Firefox/27.0")
            plugin command("SocketCommands.dll", "socket set header", "Referer", "URL")
            plugin command("SocketCommands.dll", "socket navigate", "GET", #QueryURL)
            set(#HTML, $plugin function("SocketCommands.dll", "$socket page html"), "Local")
            if($contains(#HTML, "Record ID: {#COUNT}")) {
                then {
                    set(#Email, $find regular expression($plugin function("SocketCommands.dll", "$socket page html"), "(?<=mailto:).*?(?=\")"), "Local")
                    set(#Name, $find regular expression($plugin function("SocketCommands.dll", "$socket page html"), "(?<=Name</b></td>\\r\\n\\s\\s\\s\\s\\s\\s\\s\\s\\s\\s\\s\\s<td">).*?(?=</td>)"), "Local")
                    set table cell(&RESULTS, #RESULTS ROW, 0, #COUNT)
                    set table cell(&RESULTS, #RESULTS ROW, 1, #Email)
                    set table cell(&RESULTS, #RESULTS ROW, 2, #Name)
                }
                else {
                    set(#Email, "Failed", "Local")
                    set(#Name, "Failed", "Local")
                    set table cell(&RESULTS, #RESULTS ROW, 0, #COUNT)
                    set table cell(&RESULTS, #RESULTS ROW, 1, #Email)
                    set table cell(&RESULTS, #RESULTS ROW, 2, #Name)
                }
            }
        }
        comment("Decrement Number Of Threads And Store Value to UBot Variable.")
        set(#THREADS Active, $plugin function("Threads Counter.dll", "threads counter", "decrement"), "Global")
    }
}
comment("Wait For Threads To Close Or Timeout After 20 seconds.")
set(#THREADS Active, $plugin function("Threads Counter.dll", "threads counter", "read"), "Global")
loop while($comparison(#THREADS Active, ">", 0)) {
    set(#THREADS Active, $plugin function("Threads Counter.dll", "threads counter", "read"), "Global")
    wait(2)
}
comment("Check How Many Threads UBot Sees.")
if($comparison(#THREADS Active, ">", 0)) {
    then {
        alert("Some threads were not closed/decremented properly.")
    }
    else {
        alert("All threads were closed")
    }
}
save to file("path", &RESULTS)
stop script

If anyone has any bright ideas, I'd sincerely appreciate it.

Link to post
Share on other sites
  • 2 weeks later...

Dumb question, I purchased v5 and so have never had v4 - can I get it somewhere? I submitted a support ticket, but thought I'd check here. Appreciate the help.

I think they don't offer the downloads anymore, so contacting them via support ticket is the right way.

Link to post
Share on other sites

I would like to know what was the maximum number of threads you run without problems. At high number like 100 or 200 threads, threading get corrupt. Bot hangs out at 200/200 but new browsers and threads are not visible in sidebar... (in other words it freeze)

 

Any solution for this?

Link to post
Share on other sites

I would like to know what was the maximum number of threads you run without problems. At high number like 100 or 200 threads, threading get corrupt. Bot hangs out at 200/200 but new browsers and threads are not visible in sidebar... (in other words it freeze)

 

Any solution for this?

 

you run 200 threads using new browser node? wow

 

my max number on full browser based multithread is 10. above that mostly unstable

  • Like 1
Link to post
Share on other sites

I would like to know what was the maximum number of threads you run without problems. At high number like 100 or 200 threads, threading get corrupt. Bot hangs out at 200/200 but new browsers and threads are not visible in sidebar... (in other words it freeze)

 

Any solution for this?

I've successfully ran 100 threads without any problems. When you run 200 threads, have you checked if your computer can handle it? Check RAM,CPU usage and if they are on 100% all the time the bot will become unresponsive... If that's the case you could try with a better computer...

Link to post
Share on other sites

Yea, my comp (server) is very strong and when running bot i run at 30% with 100 threads. but i think jamesfar is saying, if you use in browser inside thread, then threads become unstable and we have problems and browser starts crashing and i think when browser crash, thread crash as well. So i can say that around 20 threads i dont see so much crashing. So maybe like 15 threads is OK. But to be honest that is kind of low. So any ideas how to increase that?

Link to post
Share on other sites

Yea, my comp (server) is very strong and when running bot i run at 30% with 100 threads. but i think jamesfar is saying, if you use in browser inside thread, then threads become unstable and we have problems and browser starts crashing and i think when browser crash, thread crash as well. So i can say that around 20 threads i dont see so much crashing. So maybe like 15 threads is OK. But to be honest that is kind of low. So any ideas how to increase that?

 

Perhaps runs on multiple computers or vps

Link to post
Share on other sites

Yea, my comp (server) is very strong and when running bot i run at 30% with 100 threads. but i think jamesfar is saying, if you use in browser inside thread, then threads become unstable and we have problems and browser starts crashing and i think when browser crash, thread crash as well. So i can say that around 20 threads i dont see so much crashing. So maybe like 15 threads is OK. But to be honest that is kind of low. So any ideas how to increase that?

Till today I only had one bot which was crashing constantly when running on 100 threads, and the solution was to close the bot after a while and re-open and restart it from where where it left off. This way I was able to process millions of inputs without any problem... 

  • Like 1
Link to post
Share on other sites
  • 1 month later...

Very Good plugin. Thank you.

 

1 Question

 

Aymen Http Post Plugin  Http Page Scraper, $http get or $http post command  proxy change don't stable.

 

Problem Solve? help me?

 

Thank you.

Link to post
Share on other sites

Very Good plugin. Thank you.

 

1 Question

 

Aymen Http Post Plugin  Http Page Scraper, $http get or $http post command  proxy change don't stable.

 

Problem Solve? help me?

 

Thank you.

I don't think that's related to this plugin, since threads have nothing to do with HTTP POST plugin. So you should better ask Aymen if that plugin is not stable.

Link to post
Share on other sites

I don't think that's related to this plugin, since threads have nothing to do with HTTP POST plugin. So you should better ask Aymen if that plugin is not stable.

 

 

your plugin very Good dear, realy working. :)

 

http://kilifcenter.com/proxy.png

 

 

 

Problem = http post plugin

Link to post
Share on other sites

This works really well when using lower threads, but when using my proxy checker @ 100+ threads, the count goes way off and ends up at 0 even though the process of checking proxies is far from complete. 

Link to post
Share on other sites

This works really well when using lower threads, but when using my proxy checker @ 100+ threads, the count goes way off and ends up at 0 even though the process of checking proxies is far from complete. 

Are you sure it's not your code? Paste it here is you want us to help you. I haven't had problems since using it, so I'm not sure what could be going wrong there...

Link to post
Share on other sites

I used the plugin for one of my scraping bots with 100+ threads. And haven't had a single issue with it.

 

If your "problem" is, that the active threads is lower than what you have configured as your target number, you could try to lower the wait time in this section:

 

loop while($comparison($plugin function("Threads Counter.dll", "threads counter", "read"), ">=", 100)) {
        wait(0.5)
    }

 

The more threads you run, the more impact this wait command has. So if that is set too high, it could result in an average of 94 threads instead of 100 threads. 

 

Bu this is not an issue. It's just that some threads are finishing while the wait command is still executing.

 

Dan

Link to post
Share on other sites

I used the plugin for one of my scraping bots with 100+ threads. And haven't had a single issue with it.

 

There is no problem with the threads counter plugin. But I am sure if you are using threads along with add to list there are problems with your scraping. The issue is even though the thread counter helps there is still a problem when adding data to a list at the exact same time as another thread. Run my code and you will see that not all of the items are being added to the list when running threads.

Threads and Add Items (test).ubot

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...