Jump to content
UBot Underground

MUST READ: Threading Doesn't Work As Expected (tested in v4)


Recommended Posts

I was working on a project where threads started to disappear when bot processed a large number of items. The bot was running 50 threads and when it processed a few thousand items only a few threads out of 50 were still running or even only one in extreme cases, but the global "active threads" variable/counter still showed 50 threads running (the value was actually fluctuating from 50 to 47, if 3 threads/browsers were still running for example). Consequentially when the bot was about to finish, the "active threads" value didn't drop to 0 as expected so the bot stayed in the last "loop while" statement forever, where it was waiting for active threads to drop to 0 (although there were no active browsers running and according to code that value should be 0 when bot reaches the end).

 

To try and solve this problem I've created a "dummy" test code which does noting, only waits and spawns threads, and I've found out that the same thing happens (http://screencast.com/t/cX0x6hRS239z), when "active threads" counter should drop to 0 it didn't, the value of counter was left at the random value, so I soon realized that global variable doesn't get decremented as many times as it gets incremented.

 

PROBLEM:

The thing is that when we decrement a variable value the value is first read, then decremented and at last stored back into variable. This works consistent if we are decrementing that value from one thread, but it doesn't work as expected if two threads are decrementing global variable at the same time, because both threads will read the same value, decrement it and then store it back to the variable....so instead that global variable value would be decremented by 2, it gets decremented by 1, because 2nd thread overrides the value written from the 1st thread (which ended earlier).

 

Bellow is the code that I used for testing (the problem described above is repeatable with this code):

ui drop down("COUNT", "100,1000,10000", #Input Count)
ui drop down("Threads", "10,20,30,40,50", #INPUT Threads)
ui stat monitor("Threads (Active/Total):", "<b>{#THREADS Active}/{#THREADS Max}</b>")
ui stat monitor("COUNT:", #COUNT)
set(#COUNT, #Input Count, "Global")
set(#THREADS Max, #INPUT Threads, "Global")
set(#THREADS Active, 0, "Global")
loop while($comparison(#COUNT, ">", 0)) {
    if($comparison(#THREADS Active, "<", #THREADS Max)) {
        then {
            increment(#THREADS Active)
            THREAD START()
            decrement(#COUNT)
        }
        else {
            wait(0.2)
        }
    }
}
set(#WAIT Count, 0, "Global")
loop while($both($comparison(#THREADS Active, ">", 0), $comparison(#WAIT Count, "<", 10))) {
    increment(#WAIT Count)
    wait(2)
}
if($comparison(#THREADS Active, ">", 0)) {
    then {
        alert("Some threads were not closed/decremented properly.")
    }
    else {
        alert("All threads were closed")
    }
}
define THREAD START {
    thread {
        wait(0.5)
        decrement(#THREADS Active)
    }
}

SOLUTION:

First I wanted to see if there is a way to solve this in UBot, but I couldn't find it (I had a few ideas to use lists and tables, but it looks like all behave the same), so I've decided to create a plugin that provides a counter which can be changed from multiple threads at the same time (I will publish it as soon as I get plugin key from support, which I already requested). I've already tested the plugin and the problem I describe above disappeared on the project I was working on; when bot processed all items "active threads" counter fell back to 0 as soon as last browser closed and bot stopped. Here you can see plugin in action inside that dummy bot: http://screencast.com/t/xIy6HW4Ps04 (you can compare this video to the one above, the only difference between codes is that I use "threads counter" from new plugin in this 2nd video).

 

 

However, I wrote this post because I think most of us use this approach for threading and there are a lot of examples that present this as solution. It wasn't easy to make such claim and I was hesitating for some time, but after I've tested threads inside that dummy bot I've decided to share my findings here Did anyone notice similar behavior? Can someone confirm this problem?

 

I think it worked for me (and others) till today, because I was using larger delays, less threads and less items to process in previous bots, so the chance for that problem was smaller than it is with "faster" bots.

 

EDIT: I've released the plugin which fixes this issue here: http://www.ubotstudio.com/forum/index.php?/topic/15441-free-plugin-threads-counter-ubot-v4-threading-fixed/

  • Like 6
Link to post
Share on other sites
  • Replies 56
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

I was working on a project where threads started to disappear when bot processed a large number of items. The bot was running 50 threads and when it processed a few thousand items only a few threads o

Thanks for reporting your findings to the community. There is definitely a problem with the threads remaining open because of the decrement command not reducing the number of threads running. I have n

Nice to hear that I'm not the only one experiencing this. I never noticed someone mentioning this problems and it somehow worked for me till today, that's why I wasn't sure if this really is a problem

Thanks for reporting your findings to the community. There is definitely a problem with the threads remaining open because of the decrement command not reducing the number of threads running. I have noticed this happening for a while and my only solution was to add a button in my bot where the user would have to manually decrement a thread. However, this was not a real solution at all. So paid or free I look forward to obtaining your plugin.

  • Like 1
Link to post
Share on other sites

Thanks for reporting your findings to the community. There is definitely a problem with the threads remaining open because of the decrement command not reducing the number of threads running. I have noticed this happening for a while and my only solution was to add a button in my bot where the user would have to manually decrement a thread. However, this was not a real solution at all. So paid or free I look forward to obtaining your plugin.

Nice to hear that I'm not the only one experiencing this. I never noticed someone mentioning this problems and it somehow worked for me till today, that's why I wasn't sure if this really is a problem.

 

I agree...we can't expect final users to do that. :)

 

The plugin will be free and I'll release it as soon as I get the key and prepare the page on my blog.

  • Like 1
Link to post
Share on other sites

Nice to hear that I'm not the only one experiencing this. I never noticed someone mentioning this problems and it somehow worked for me till today, that's why I wasn't sure if this really is a problem.

 

I agree...we can't expect final users to do that. :)

 

The plugin will be free and I'll release it as soon as I get the key and prepare the page on my blog.

 

Well what I noticed was that my threads would remain open and when I would check the debugger it was that my running thread count wasn't decremented. But thanks to your post I now know the reason is because of the global variable not being decremented when the command is ran at the same exact time. Because your right, its not that the threads hang all of the time. Just when you run a greater number of threads is where it becomes a problem.

  • Like 1
Link to post
Share on other sites

Well what I noticed was that my threads would remain open and when I would check the debugger it was that my running thread count wasn't decremented. But thanks to your post I now know the reason is because of the global variable not being decremented when the command is ran at the same exact time. Because your right, its not that the threads hang all of the time. Just when you run a greater number of threads is where it becomes a problem.

I couldn't find any other reasons....it was clear to me that "thread" commands were already executed since there were no browsers, that's why I started digging deeper. Yeah, I also think that problem kicks in only if you are using a lot of threads and a lot of "items" to process, since the chance that 2 threads change global variable at the same time is greater in that case.

  • Like 1
Link to post
Share on other sites

Current version specifically solves the problem for cases like one above (threading with one counter); it provides one global counter which supports atomic operations.

 

The problem doesn't exist for all shared variables, only for the ones that get modified (read itself isn't problematic), so you could be right for that part.

 

However, I'm running an extensive test (140k items, 50 threads) and the bot is saving scraped data into a global table. I've checked the first 10k results and it looks like all the data was stored properly - all rows are there as expected, so I think that UBot table could be threads friendly).

 

If adding cells to table would be failing I think I would see some empty cells where I shouldn't, but I don't, at lest for now. Because the bot uses more "set table" commands (10) than it does "decrement"  "threads counter" commands (1), so I think it's safe to say that UBot tables support atomic operations (probability for error with "set table cell" is 10 times higher but no errors). I will check the other files later when bot stops running to see if table got corrupted and will report back later.

Link to post
Share on other sites

Yeah, I too have noticed same type of threads not decrementing (closing).

 

 

I was going to start a thread with what I found, after talking with some peeps in Skype group but you beat me to it so Iwill just add here only to aid in plugin development.

 

OK, we know RAM builds up in browser,exe "Memory leak" and I knew the threads weren't closing.

 

This how I am dealing with it now.

 

Pretty much bots run fine 10-20 threads for about 20-30 minutes depending on the machine.

 

So I close bot after it's done with task and then open new one with onload. Simple right? then it just runs all day.

 

I know the task will take 20-30 minutes just from running  previous test. Delay and number of accounts in list etc.

 

But really i just want one bot to run all day, as I'm sure the same is true with all of you.

 

So memory starts fresh and threads start fresh and essentially I have a marathon bot.

 

I'm like hmm...threads must not be closing. So I start playing around and I found that if you decrease threads manually and wait till threads are all closed then increase threads again the RAM drops and slowly builds over time again. I use text box and do not set threads to 0, I set at 2 then wait. So now I am left at how do I automate that?

 

This part is theory and have not tested yet. I figure if I can do it manually I can do it programmatically right?

 

So I came up with loop while and every 20 min. change my thread variable to 2 until open threads equal 2 and then set threads variable back to original. Or something like this. Eventually I will try it.

 

I hope this helps and my intention is not to take over thread and just give my perspective on this threads issue.

 

So if you are not using something like this or some variation maybe you could incorporate this into your plugin.

 

Like if RAM exceeds 80% (user defined) reduce threads to 1 or 2 for X amount of time or like I said above, when used threads = 2

 

 

That's how I make sure those damn threads close.

 

Thanks for sharing your plugins!!!!

UBotDev.com

 

TC

  • Like 1
Link to post
Share on other sites

staff is aware and i submitted similar test code for them

 

in dev view you can switch to debug options and it will work as its suppose to but not in debug mode it will not work properly.

this is with cross thread sharing of the variables for thread tracking.

 

and causes them to get thrown off and not work properly.

Link to post
Share on other sites

Just to let you know...the test I was running never completed...it got to 80k items processed but then the bot stopped with one thread and browser opened: http://screencast.com/t/JuCSvoHmT7sZ (counter working properly now). That browser doesn't want to close itself so the bot is stuck there. I've also noticed that bot.exe is using all of the available memory, even when I save data from table to file and clear that table, so I also think there is a memory leak. Regarding the "browser.exe"...there are 4 processes still running, and they all use small amount of memory.

Link to post
Share on other sites

Just to let you know...the test I was running never completed...it got to 80k items processed but then the bot stopped with one thread and browser opened: http://screencast.com/t/JuCSvoHmT7sZ (counter working properly now). That browser doesn't want to close itself so the bot is stuck there. I've also noticed that bot.exe is using all of the available memory, even when I save data from table to file and clear that table, so I also think there is a memory leak. Regarding the "browser.exe"...there are 4 processes still running, and they all use small amount of memory.

I tried writing to a table with 10000 rows, 100 threads and all the tables cells where filled correctly so can confirm tables can handles multi operations. not sure about your issue looks like the normal memory leek issue.

Link to post
Share on other sites

Pretty much bots run fine 10-20 threads for about 20-30 minutes depending on the machine.

 

So I close bot after it's done with task and then open new one with onload. Simple right? then it just runs all day.

I also think I'll have to do that if browser will keep crashing (I think that was the case in test that I ran)

 

 

 

I'm like hmm...threads must not be closing. So I start playing around and I found that if you decrease threads manually and wait till threads are all closed then increase threads again the RAM drops and slowly builds over time again. I use text box and do not set threads to 0, I set at 2 then wait. So now I am left at how do I automate that?

 

This part is theory and have not tested yet. I figure if I can do it manually I can do it programmatically right?

You could do that programatically....at the end you would have a "loop while" command, which would be waiting for threads to close, but if there are active threads a few minutes after the bot reached the loop you could programatically decrement active threads, to get them down to 0. However I think once you have the plugin this won't happen anymore...I've tested it yesterday with Gogetta  and it seemed like problem was solved.

 

 

I hope this helps and my intention is not to take over thread and just give my perspective on this threads issue.

 

So if you are not using something like this or some variation maybe you could incorporate this into your plugin.

 

Like if RAM exceeds 80% (user defined) reduce threads to 1 or 2 for X amount of time or like I said above, when used threads = 2

 

No problem, I opened this thread so we can discuss. I don't think I'll incorporate that into the plugin for now...

 

 

I tried writing to a table with 10000 rows, 100 threads and all the tables cells where filled correctly so can confirm tables can handles multi operations. not sure about your issue looks like the normal memory leek issue.

Nice to hear. Yeah, me neither...it looks like that remaining browser didn't even load HTML, since it has the same background color as UBot does, so I think something went wrong when the browser got opened. :/

Link to post
Share on other sites

I also have that 1 remaining browser and thread then bot won't finish and go to next command like exit program.so next bot can open...

 

It doesn't happen all the time tho.

 

 

ahhhh!!

 

 

 

Some day!! guys

Link to post
Share on other sites

staff is aware and i submitted similar test code for them

 

in dev view you can switch to debug options and it will work as its suppose to but not in debug mode it will not work properly.

this is with cross thread sharing of the variables for thread tracking.

 

and causes them to get thrown off and not work properly.

Yea, and thats cool and everything but they have a long list too.

 

I think of plugin developers as like threads, more threads...more stuff gets done ;)

Link to post
Share on other sites

This may help you guys, but I have not used it at 50 threads

set(#BrowserCount, $BrowserCount(), "Global")
define $BrowserCount {
    clear list(%Browsers)
    add list to list(%Browsers, $list from text($plugin function("Advanced Shell.dll", "$shell batch hidden", "tasklist /fi \"imagename eq Browser.exe"), $new line), "Delete", "Local")
    loop(2) {
        remove from list(%Browsers, 0)
    }
    return($list total(%Browsers))
}

 

Link to post
Share on other sites

This may help you guys, but I have not used it at 50 threads

set(#BrowserCount, $BrowserCount(), "Global")
define $BrowserCount {
    clear list(%Browsers)
    add list to list(%Browsers, $list from text($plugin function("Advanced Shell.dll", "$shell batch hidden", "tasklist /fi \"imagename eq Browser.exe"), $new line), "Delete", "Local")
    loop(2) {
        remove from list(%Browsers, 0)
    }
    return($list total(%Browsers))
}

 

 

I miss-read that first.

 

I don't think relying on number of browser from task manager is the best way, since there are still cases where your code would fail...for example if browser.exe doesn't get closed or if you have more than 2 browsers initially (2 UI's for example).

 

Btw, I'm still waiting for the plugin key...I got reply roday but they didn't send the key .:/ Now I'm waiting again...

Link to post
Share on other sites

Sorry I should of explained and not just left you guys to work it out, this is not to control the threads but to take a snapshot of the browser count and also the PID for each before you start multithreading, allowing you to reset the number of threads/browsers after x number of cycles, if you look at the code it can return both the browser count the PID, This gives you the baseline to work from, using this and fractions within your delays worked pretty well for me, as this problem has been about from version 3.5.
But as you don’t need any delays with sockets I’m not sure it would work so well 

Link to post
Share on other sites
  • 2 weeks later...

 

allowing you to reset the number of threads/browsers after x number of cycles,

 

zap? could you kindly elaborate on how we could achieve this? some sample code please?

Thanks.

Link to post
Share on other sites

I've been trying out ways to fix this for days. Some ideas have been far better than others. But so far this is the fastest while being the most accurate. 

 

Test:

The original code at 1000 count and 30 threads. Result: 7+ minutes and 29 threads not closing (29 dead threads as it becomes apparent half way through when the numbers start dropping 1 by 1)

 

This is my solution and the result is much faster as it should be and the result is 1 dead thread at the end.

 

I would love to hear your thoughts and ideas on how to make it better and I can't wait for that plugin UbotDev please release it :D

Edit: More results:
 

10000 count 40 threads - 17 dead
10000 count 50 threads - 20 dead
 
1000 count 10 threads - 2 dead
1000 count 20 threads - 2 dead
1000 count 30 threads - 1 dead
1000 count 40 threads - 0 dead
ui drop down("COUNT", "100,1000,10000", #Input Count)
ui drop down("Threads", "10,20,30,40,50", #INPUT Threads)
ui stat monitor("Threads (Active/Total):", "<b>{#THREADS Active}/{#THREADS Max}</b>")
ui stat monitor("COUNT:", #COUNT)
set(#COUNT, #Input Count, "Global")
set(#THREADS Max, #INPUT Threads, "Global")
clear list(%threads)
set(#THREADS Active, 0, "Global")
loop while($comparison(#COUNT, ">", 0)) {
    if($comparison($list total(%threads), "<", #THREADS Max)) {
        then {
            add item to list(%threads, 1, "Don\'t Delete", "Global")
            set(#THREADS Active, $list total(%threads), "Global")
            THREAD START()
            decrement(#COUNT)
        }
        else {
            wait(0.2)
        }
    }
}
set(#WAIT Count, 0, "Global")
loop while($both($comparison(#THREADS Active, ">", 0), $comparison(#WAIT Count, "<", 10))) {
    set(#THREADS Active, $list total(%threads), "Global")
    increment(#WAIT Count)
    wait(2)
}
if($comparison(#THREADS Active, ">", 0)) {
    then {
        alert("Some threads were not closed/decremented properly.")
    }
    else {
        alert("All threads were closed")
    }
}
define THREAD START {
    thread {
        wait(0.5)
        remove from list(%threads, 0)
    }
}

Link to post
Share on other sites

I did a table that on start up would  create the number of rows in the table based on the number of threads you have set.

when a thread started it would write "used" in that rows table cell it would then pass the row number to the define with your code you want to run in it. when the define had finished it would write free in that row. the thread control worked using table search for a free slot it looped until it found one it.  this worked incredibly well the only issue was ubot never finished you had to use stop script command(I think this was because of the amount of writing to variables it was doing in a short space of time.

it was accurate at 300 threads didn't go any higher

 

have a play if anyone is interested.

define thread contol(#max threads, #numberofloops) {
    clear table(&threads)
    set(#startupthreads, 0, "Global")
    loop(#numberofloops) {
        if($comparison($table total rows(&threads), "<", #max threads)) {
            then {
                set table cell(&threads, #startupthreads, 0, "used")
                threadcode(#startupthreads)
                increment(#startupthreads)
            }
            else {
                set(#assigned, 0, "Global")
                loop while($comparison(#assigned, "=", 0)) {
                    set(#threadsearch, $plugin function("TableCommands.dll", "$table search", &threads, "Free", "Row Index"), "Global")
                    if($comparison(#threadsearch, ">=", 0)) {
                        then {
                            set table cell(&threads, #threadsearch, 0, "used")
                            threadcode(#threadsearch)
                            set(#assigned, 1, "Global")
                        }
                        else {
                            wait(1)
                        }
                    }
                }
            }
        }
    }
}
comment("Put your code in threadcode")
define threadcode(#threadnumber) {
    thread {
        wait($rand(10, 20))
        set table cell(&threads, #threadnumber, 0, "Free")
    }
}
thread contol(150, 1000)
set(#allthreadsclosed, $plugin function("TableCommands.dll", "$table search", &threads, "used", "Row Index"), "Global")
loop while($comparison(#allthreadsclosed, ">=", 0)) {
    wait(0.25)
    set(#allthreadsclosed, $plugin function("TableCommands.dll", "$table search", &threads, "used", "Row Index"), "Global")
}
alert("all threads closed")
stop script
  • Like 1
Link to post
Share on other sites

I've also tried to solve this problem with UBot list, but results were the same...so lists aren't thread safe.

 

As mentioned before tables could be thread safe (I assume that because no data was lost in extended test), so the solution Kevin provided above could work well.

 

Regarding the plugin...I'm still waiting for key and I'm not sure what's taking so long. Maybe they are just going to fix this on their own.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...