Jump to content



Photo

MUST READ: Threading Doesn't Work As Expected (tested in v4)


  • Please log in to reply
56 replies to this topic

#21 Sanjeev

Sanjeev

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 269 posts
  • LocationIndia, Mumbai
  • OS:Windows 10
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 24 October 2013 - 10:28 PM

 

allowing you to reset the number of threads/browsers after x number of cycles,

 

zap? could you kindly elaborate on how we could achieve this? some sample code please?

Thanks.



#22 HelloInsomnia

HelloInsomnia

    Advanced Member

  • Moderators
  • 2301 posts
  • OS:Windows 10
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 26 October 2013 - 01:14 PM

I've been trying out ways to fix this for days. Some ideas have been far better than others. But so far this is the fastest while being the most accurate. 

 

Test:

The original code at 1000 count and 30 threads. Result: 7+ minutes and 29 threads not closing (29 dead threads as it becomes apparent half way through when the numbers start dropping 1 by 1)

 

This is my solution and the result is much faster as it should be and the result is 1 dead thread at the end.

 

I would love to hear your thoughts and ideas on how to make it better and I can't wait for that plugin UbotDev please release it :D

Edit: More results:
 

10000 count 40 threads - 17 dead
10000 count 50 threads - 20 dead
 
1000 count 10 threads - 2 dead
1000 count 20 threads - 2 dead
1000 count 30 threads - 1 dead
1000 count 40 threads - 0 dead
ui drop down("COUNT", "100,1000,10000", #Input Count)
ui drop down("Threads", "10,20,30,40,50", #INPUT Threads)
ui stat monitor("Threads (Active/Total):", "<b>{#THREADS Active}/{#THREADS Max}</b>")
ui stat monitor("COUNT:", #COUNT)
set(#COUNT, #Input Count, "Global")
set(#THREADS Max, #INPUT Threads, "Global")
clear list(%threads)
set(#THREADS Active, 0, "Global")
loop while($comparison(#COUNT, ">", 0)) {
    if($comparison($list total(%threads), "<", #THREADS Max)) {
        then {
            add item to list(%threads, 1, "Don\'t Delete", "Global")
            set(#THREADS Active, $list total(%threads), "Global")
            THREAD START()
            decrement(#COUNT)
        }
        else {
            wait(0.2)
        }
    }
}
set(#WAIT Count, 0, "Global")
loop while($both($comparison(#THREADS Active, ">", 0), $comparison(#WAIT Count, "<", 10))) {
    set(#THREADS Active, $list total(%threads), "Global")
    increment(#WAIT Count)
    wait(2)
}
if($comparison(#THREADS Active, ">", 0)) {
    then {
        alert("Some threads were not closed/decremented properly.")
    }
    else {
        alert("All threads were closed")
    }
}
define THREAD START {
    thread {
        wait(0.5)
        remove from list(%threads, 0)
    }
}


eLBF9bO.png s7pbT0S.png


#23 kev123

kev123

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 735 posts
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v3.5 & v4.0
  • License:Developer Edition

Posted 26 October 2013 - 02:10 PM

I did a table that on start up would  create the number of rows in the table based on the number of threads you have set.

when a thread started it would write "used" in that rows table cell it would then pass the row number to the define with your code you want to run in it. when the define had finished it would write free in that row. the thread control worked using table search for a free slot it looped until it found one it.  this worked incredibly well the only issue was ubot never finished you had to use stop script command(I think this was because of the amount of writing to variables it was doing in a short space of time.

it was accurate at 300 threads didn't go any higher

 

have a play if anyone is interested.

define thread contol(#max threads, #numberofloops) {
    clear table(&threads)
    set(#startupthreads, 0, "Global")
    loop(#numberofloops) {
        if($comparison($table total rows(&threads), "<", #max threads)) {
            then {
                set table cell(&threads, #startupthreads, 0, "used")
                threadcode(#startupthreads)
                increment(#startupthreads)
            }
            else {
                set(#assigned, 0, "Global")
                loop while($comparison(#assigned, "=", 0)) {
                    set(#threadsearch, $plugin function("TableCommands.dll", "$table search", &threads, "Free", "Row Index"), "Global")
                    if($comparison(#threadsearch, ">=", 0)) {
                        then {
                            set table cell(&threads, #threadsearch, 0, "used")
                            threadcode(#threadsearch)
                            set(#assigned, 1, "Global")
                        }
                        else {
                            wait(1)
                        }
                    }
                }
            }
        }
    }
}
comment("Put your code in threadcode")
define threadcode(#threadnumber) {
    thread {
        wait($rand(10, 20))
        set table cell(&threads, #threadnumber, 0, "Free")
    }
}
thread contol(150, 1000)
set(#allthreadsclosed, $plugin function("TableCommands.dll", "$table search", &threads, "used", "Row Index"), "Global")
loop while($comparison(#allthreadsclosed, ">=", 0)) {
    wait(0.25)
    set(#allthreadsclosed, $plugin function("TableCommands.dll", "$table search", &threads, "used", "Row Index"), "Global")
}
alert("all threads closed")
stop script


#24 kev123

kev123

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 735 posts
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v3.5 & v4.0
  • License:Developer Edition

Posted 26 October 2013 - 02:13 PM

wait($rand(10, 20)) threadcode  is only to simulate commands it can be removed when you put your code to multhtread in threadcode



#25 UBotDev

UBotDev

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 1395 posts
  • OS:Windows 10
  • Total Memory:More Than 9Gb
  • Framework:v3.5 & v4.0
  • License:Developer Edition

Posted 26 October 2013 - 02:53 PM

I've also tried to solve this problem with UBot list, but results were the same...so lists aren't thread safe.

 

As mentioned before tables could be thread safe (I assume that because no data was lost in extended test), so the solution Kevin provided above could work well.

 

Regarding the plugin...I'm still waiting for key and I'm not sure what's taking so long. Maybe they are just going to fix this on their own.


CLICK UBotDev.com - Download FREE UBot Plugins!

______________________________________________

 


#26 TheBigWeb

TheBigWeb

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 196 posts
  • OS:Windows 7
  • Total Memory:6Gb
  • Framework:v4.0
  • License:Developer Edition

Posted 26 October 2013 - 03:32 PM

Try this free plugin. Killed this problem dead in its tracks and works every time :)

http://www.ubotstudi...safe-container/

#27 UBotDev

UBotDev

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 1395 posts
  • OS:Windows 10
  • Total Memory:More Than 9Gb
  • Framework:v3.5 & v4.0
  • License:Developer Edition

Posted 26 October 2013 - 06:23 PM

Try this free plugin. Killed this problem dead in its tracks and works every time :)

http://www.ubotstudi...safe-container/

I don't think that will help, because the problem is not with container, rather with UBot command "increment", "set variable", "add item to list",....


CLICK UBotDev.com - Download FREE UBot Plugins!

______________________________________________

 


#28 Sanjeev

Sanjeev

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 269 posts
  • LocationIndia, Mumbai
  • OS:Windows 10
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 27 October 2013 - 01:58 AM

Could this be considered a slow but very stable solution?

Limit multithreading to say max 10-15 threads? - (while running, do regular memory cleaning + when the bot starts, set a timer - maybe on load?- this is to shut the bot down after say 20 minutes.-the bot will check if this target has been reached or not - maybe via a separate thread...?)
Also perform  'close page' for 'new browsers' -  at the end of code block.
After 20 minutes - save data + close bot + optionally before shut down ,let the main program 'open' a previously compiled  bot ->which would just clean memory (os addin plugin?)- and restart the main bot and then shut itself down.
If adding delay helps in threads - how much and where should we add them?



#29 TheBigWeb

TheBigWeb

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 196 posts
  • OS:Windows 7
  • Total Memory:6Gb
  • Framework:v4.0
  • License:Developer Edition

Posted 27 October 2013 - 02:59 AM

I don't think that will help, because the problem is not with container, rather with UBot command "increment", "set variable", "add item to list",....


It definitely worked for me. I was having the exact same problem as described. Here is how I used it: -

when you increment the global thread count varable put it inside a 'thread safe container'. This will force all other threads to wait until the code within the container has executed. You also need to put reading values and writing values to global shared variables and lists within this container. This prevents multiple threads from reading and writing the same values. If you do this properly you can keep track of all the threads.

As an extra precaution, I put an if statement when I increment and decrement the thread counter to ensure that it does not exceed the max threads variable before I increment it and above zero before decrement it . These commands are done within a thread safe container. Here is an example of my thread control code: -

 

Remember, you will need the 'Thread Safe Container Plugin' http://www.sendspace.com/file/f0cnbx

ui drop down("Threads", "1,2,3,4,5,6,7,8,9,10,15,20,25,30,35,40,45,50", #uiThreadsMAX)
ui button("Do Some Stuff") {
    if($not(#isRunning)) {
        then {
            set(#isRunning, $true, "Global")
            threadsRun()
        }
        else {
            alert("Please wait for submission to finish!")
        }
    }
}
ui stat monitor("Active Threads: ", #threadsActive)
divider
define _threadsInitialise {
    comment("********** Set thread count **********")
    set(#threadCount, #uiThreadsMAX, "Global")
    comment("********** Reset active thread count **********")
    set(#threadsActive, 0, "Global")
}
define _threadsWait {
    comment("********** Wait for all threads to finish **********")
    comment("********** Give time for threads to fire up **********")
    wait(5)
    loop while($comparison(#threadsActive, ">", 0)) {
        wait(1)
    }
}
define _threadRun {
    plugin command("Open.Framework.dll", "Thread safe container") {
        if($comparison(#threadsActive, "<", #uiThreadsMAX)) {
            then {
                increment(#threadsActive)
                comment("********** Example safe global variable (LOCAL) **********")
                set(#safe, #threadsActive, "Local")
            }
        }
    }
    in new browser {
        comment("********** Fake some code **********")
        navigate("http://google.com/", "Wait")
        type text(<name="q">, "This is a search from thread {#safe}", "Standard")
        comment("********** Fake some delay below **********")
        wait($rand(5, 10))
    }
    divider
    comment("********** Kill new thread and track **********")
    plugin command("Open.Framework.dll", "Thread safe container") {
        if($comparison(#threadsActive, ">", 0)) {
            then {
                decrement(#threadsActive)
            }
        }
    }
}
divider
define threadsRun {
    comment("********** Initialise threads **********")
    _threadsInitialise()
    comment("********** Run threads **********")
    loop(#threadCount) {
        thread {
            _threadRun()
        }
    }
    comment("********** Wait for all threads to finish **********")
    _threadsWait()
    comment("********** Set state to stopped **********")
    set(#isRunning, $false, "Global")
    alert("Submission finished!")
}

The above combination has completely eliminated the problem for me and works like a charm :)

Attached Files



#30 kev123

kev123

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 735 posts
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v3.5 & v4.0
  • License:Developer Edition

Posted 27 October 2013 - 07:34 AM

thebigweb example does work i'm doing something similar for my browser bots and low thread http bots.

 

but when you come to multithreading http over a hundred it still has issues. I believe there are two issues here

 

1.variables not being thread safe.

2.variables not updating quick enough and counts being lost

 

as you can see even with the thread safe container go over a certain amount of threads with a lot of cycles and ubot misses some decrements

 

example you will need the plugin linked in previous post

 

you can even drop the thread count down to around 100 and the issue is still there

set(#usedthreads, 0, "Global")
set(#maxthreads, 300, "Global")
loop(6000) {
    loop while($comparison(#usedthreads, ">=", #maxthreads)) {
        wait(1)
    }
    plugin command("Open.Framework.dll", "Thread safe container") {
        increment(#usedthreads)
    }
    thread {
        wait($rand(1, 5))
        plugin command("Open.Framework.dll", "Thread safe container") {
            decrement(#usedthreads)
        }
    }
}




#31 HelloInsomnia

HelloInsomnia

    Advanced Member

  • Moderators
  • 2301 posts
  • OS:Windows 10
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 27 October 2013 - 09:56 AM

Could this be considered a slow but very stable solution?

Limit multithreading to say max 10-15 threads? - (while running, do regular memory cleaning + when the bot starts, set a timer - maybe on load?- this is to shut the bot down after say 20 minutes.-the bot will check if this target has been reached or not - maybe via a separate thread...?)
Also perform  'close page' for 'new browsers' -  at the end of code block.
After 20 minutes - save data + close bot + optionally before shut down ,let the main program 'open' a previously compiled  bot ->which would just clean memory (os addin plugin?)- and restart the main bot and then shut itself down.
If adding delay helps in threads - how much and where should we add them?

 

While that would help overall with the performance of a bot - especially one running long term. It won't help with the threading issue because they die out incredibly fast. You may start with 10 threads and be down to 1 a minute later.


eLBF9bO.png s7pbT0S.png


#32 Sanjeev

Sanjeev

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 269 posts
  • LocationIndia, Mumbai
  • OS:Windows 10
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 27 October 2013 - 10:04 AM

I don't get it?!! How do you guys sell bots if Multi threading is so badly implemented? Who would buy such bots? It doesn't make any sense..  :(

 

I don't mean to offend anyone..  I consider myself a total beginner/learner of Ubot so I am just curious how is it possible to create stable multi threaded apps for sale with so many issues...

 

Thanks.



#33 Gogetta

Gogetta

    Advanced Member

  • Moderators
  • 883 posts
  • OS:Windows 8
  • Total Memory:More Than 9Gb
  • Framework:v3.5 & v4.0
  • License:Developer Edition

Posted 27 October 2013 - 10:56 AM

I don't get it?!! How do you guys sell bots if Multi threading is so badly implemented? Who would buy such bots? It doesn't make any sense..  :(

 

I don't mean to offend anyone..  I consider myself a total beginner/learner of Ubot so I am just curious how is it possible to create stable multi threaded apps for sale with so many issues...

 

Thanks.

 

Well it wasn't really a problem when using the in browser commands since the thread count on average wouldn't be able to go higher than 5-10. But now that we are all using sockets our bots can now handle 100+. threads. Only problem now is what HelloInsomnia stated:
 

 

You may start with 10 threads and be down to 1 a minute later.

 

 

Hopefully UBotDev.com's plugin gets approved soon. I have had the privilege of testing it out and you can easily tell that the global variable problem is limiting the power of threaded bots.



#34 Steve

Steve

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 259 posts
  • OS:Windows 8
  • Total Memory:More Than 9Gb
  • Framework:v4.0
  • License:Developer Edition

Posted 28 October 2013 - 07:56 AM

Would love to see this plugin release soon. 



#35 Sanjeev

Sanjeev

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 269 posts
  • LocationIndia, Mumbai
  • OS:Windows 10
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 28 October 2013 - 08:52 AM

Are single threaded applications obsolete? would like to know what the community has to say.



#36 UBotDev

UBotDev

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 1395 posts
  • OS:Windows 10
  • Total Memory:More Than 9Gb
  • Framework:v3.5 & v4.0
  • License:Developer Edition

Posted 28 October 2013 - 09:07 AM

It definitely worked for me. I was having the exact same problem as described. Here is how I used it: -

when you increment the global thread count varable put it inside a 'thread safe container'. This will force all other threads to wait until the code within the container has executed. You also need to put reading values and writing values to global shared variables and lists within this container. This prevents multiple threads from reading and writing the same values. If you do this properly you can keep track of all the threads.

As an extra precaution, I put an if statement when I increment and decrement the thread counter to ensure that it does not exceed the max threads variable before I increment it and above zero before decrement it . These commands are done within a thread safe container. Here is an example of my thread control code: -

 

Remember, you will need the 'Thread Safe Container Plugin' http://www.sendspace.com/file/f0cnbx

 

I've tested the code you posted above with one change, which is I replaced "navigate" part with 1 second fixed delay and I noticed that that approach is not as safe as you might think.

 

It looks like plugin is writing to disk, which also fails with a lot of thread's executing fast: http://screencast.com/t/Cs2hn2wGERkd

(I get error popups which I didn't get with your code).

 

However, else it looks like all threads were closed successfully (according to this short run).

 

TEST CODE:

ui drop down("Threads", "1,2,3,4,5,6,7,8,9,10,15,20,25,30,35,40,45,50", #uiThreadsMAX)
ui button("Do Some Stuff") {
    if($not(#isRunning)) {
        then {
            set(#isRunning, $true, "Global")
            threadsRun()
        }
        else {
            alert("Please wait for submission to finish!")
        }
    }
}
ui stat monitor("Active Threads: ", #threadsActive)
define threadsRun {
    comment("********** Initialise threads **********")
    _threadsInitialise()
    comment("********** Run threads **********")
    loop(#threadCount) {
        thread {
            _threadRun()
        }
    }
    comment("********** Wait for all threads to finish **********")
    _threadsWait()
    comment("********** Set state to stopped **********")
    set(#isRunning, $false, "Global")
    alert("Submission finished!")
}
divider
define _threadsInitialise {
    comment("********** Set thread count **********")
    set(#threadCount, #uiThreadsMAX, "Global")
    comment("********** Reset active thread count **********")
    set(#threadsActive, 0, "Global")
}
define _threadRun {
    plugin command("Open.Framework.dll", "Thread safe container") {
        if($comparison(#threadsActive, "<", #uiThreadsMAX)) {
            then {
                increment(#threadsActive)
                comment("********** Example safe global variable (LOCAL) **********")
                set(#safe, #threadsActive, "Local")
            }
        }
    }
    in new browser {
        comment("********** Fake some code **********")
        wait(1)
    }
    divider
    comment("********** Kill new thread and track **********")
    plugin command("Open.Framework.dll", "Thread safe container") {
        if($comparison(#threadsActive, ">", 0)) {
            then {
                decrement(#threadsActive)
            }
        }
    }
}
define _threadsWait {
    comment("********** Wait for all threads to finish **********")
    comment("********** Give time for threads to fire up **********")
    wait(5)
    loop while($comparison(#threadsActive, ">", 0)) {
        wait(1)
    }
}
divider


CLICK UBotDev.com - Download FREE UBot Plugins!

______________________________________________

 


#37 UBotDev

UBotDev

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 1395 posts
  • OS:Windows 10
  • Total Memory:More Than 9Gb
  • Framework:v3.5 & v4.0
  • License:Developer Edition

Posted 28 October 2013 - 09:26 AM

Hopefully UBotDev.com's plugin gets approved soon. I have had the privilege of testing it out and you can easily tell that the global variable problem is limiting the power of threaded bots.

I can imagine that un-reviewed plugin popup is annoying. :)

 

Would love to see this plugin release soon. 

Still waiting for approval. :/

 

Are single threaded applications obsolete? would like to know what the community has to say.

It's not obsolete, it's just that if you have a lot of "tasks" to process you can't wait 1 year (12 months) for single threaded bot to finish (because data is usually needed asap), instead you could use 12 threads/proxies and get the job done in one month, or even use 100 threads/proxies to complete all tasks in just 4 days. Would you prefer to wait one year? :)

 

So my opinion is that single threaded applications can still be used where speed/time is not an important factor...


CLICK UBotDev.com - Download FREE UBot Plugins!

______________________________________________

 


#38 TheBigWeb

TheBigWeb

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 196 posts
  • OS:Windows 7
  • Total Memory:6Gb
  • Framework:v4.0
  • License:Developer Edition

Posted 28 October 2013 - 12:25 PM

 

I've tested the code you posted above with one change, which is I replaced "navigate" part with 1 second fixed delay and I noticed that that approach is not as safe as you might think.

 

It looks like plugin is writing to disk, which also fails with a lot of thread's executing fast: http://screencast.com/t/Cs2hn2wGERkd

(I get error popups which I didn't get with your code).

 

However, else it looks like all threads were closed successfully (according to this short run).

 

TEST CODE:

ui drop down("Threads", "1,2,3,4,5,6,7,8,9,10,15,20,25,30,35,40,45,50", #uiThreadsMAX)
ui button("Do Some Stuff") {
    if($not(#isRunning)) {
        then {
            set(#isRunning, $true, "Global")
            threadsRun()
        }
        else {
            alert("Please wait for submission to finish!")
        }
    }
}
ui stat monitor("Active Threads: ", #threadsActive)
define threadsRun {
    comment("********** Initialise threads **********")
    _threadsInitialise()
    comment("********** Run threads **********")
    loop(#threadCount) {
        thread {
            _threadRun()
        }
    }
    comment("********** Wait for all threads to finish **********")
    _threadsWait()
    comment("********** Set state to stopped **********")
    set(#isRunning, $false, "Global")
    alert("Submission finished!")
}
divider
define _threadsInitialise {
    comment("********** Set thread count **********")
    set(#threadCount, #uiThreadsMAX, "Global")
    comment("********** Reset active thread count **********")
    set(#threadsActive, 0, "Global")
}
define _threadRun {
    plugin command("Open.Framework.dll", "Thread safe container") {
        if($comparison(#threadsActive, "<", #uiThreadsMAX)) {
            then {
                increment(#threadsActive)
                comment("********** Example safe global variable (LOCAL) **********")
                set(#safe, #threadsActive, "Local")
            }
        }
    }
    in new browser {
        comment("********** Fake some code **********")
        wait(1)
    }
    divider
    comment("********** Kill new thread and track **********")
    plugin command("Open.Framework.dll", "Thread safe container") {
        if($comparison(#threadsActive, ">", 0)) {
            then {
                decrement(#threadsActive)
            }
        }
    }
}
define _threadsWait {
    comment("********** Wait for all threads to finish **********")
    comment("********** Give time for threads to fire up **********")
    wait(5)
    loop while($comparison(#threadsActive, ">", 0)) {
        wait(1)
    }
}
divider

 

Strange, I tried the modified code above but didn't receive any errors.



#39 LazyBotter

LazyBotter

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 609 posts
  • LocationManchester
  • OS:Windows 8
  • Total Memory:More Than 9Gb
  • Framework:v3.5 & v4.0
  • License:Developer Edition

Posted 14 November 2013 - 07:58 AM

Has the plugin not been approved yet???


<p>SSUBV53-e1396282301731.png

#40 UBotDev

UBotDev

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 1395 posts
  • OS:Windows 10
  • Total Memory:More Than 9Gb
  • Framework:v3.5 & v4.0
  • License:Developer Edition

Posted 14 November 2013 - 08:35 AM

No plugin key yet...I'm not sure what's taking so long...it's over a month and still no key. A few days ago they asked me if I still need the key plugin, and I said yes, else I wouldn't request it, but still no key till today. Not sure what is going on there but it looks like support is getting worse and worse.


CLICK UBotDev.com - Download FREE UBot Plugins!

______________________________________________

 





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users