Jump to content
UBot Underground

MUST READ: Threading Doesn't Work As Expected (tested in v4)


Recommended Posts

  • Replies 56
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

I was working on a project where threads started to disappear when bot processed a large number of items. The bot was running 50 threads and when it processed a few thousand items only a few threads o

Thanks for reporting your findings to the community. There is definitely a problem with the threads remaining open because of the decrement command not reducing the number of threads running. I have n

Nice to hear that I'm not the only one experiencing this. I never noticed someone mentioning this problems and it somehow worked for me till today, that's why I wasn't sure if this really is a problem

Try this free plugin. Killed this problem dead in its tracks and works every time :)

 

http://www.ubotstudio.com/forum/index.php?/topic/14205-howtorun-thread-safe-container/

I don't think that will help, because the problem is not with container, rather with UBot command "increment", "set variable", "add item to list",....

Link to post
Share on other sites

Could this be considered a slow but very stable solution?

Limit multithreading to say max 10-15 threads? - (while running, do regular memory cleaning + when the bot starts, set a timer - maybe on load?- this is to shut the bot down after say 20 minutes.-the bot will check if this target has been reached or not - maybe via a separate thread...?)
Also perform  'close page' for 'new browsers' -  at the end of code block.
After 20 minutes - save data + close bot + optionally before shut down ,let the main program 'open' a previously compiled  bot ->which would just clean memory (os addin plugin?)- and restart the main bot and then shut itself down.
If adding delay helps in threads - how much and where should we add them?

Link to post
Share on other sites

I don't think that will help, because the problem is not with container, rather with UBot command "increment", "set variable", "add item to list",....

It definitely worked for me. I was having the exact same problem as described. Here is how I used it: -

 

when you increment the global thread count varable put it inside a 'thread safe container'. This will force all other threads to wait until the code within the container has executed. You also need to put reading values and writing values to global shared variables and lists within this container. This prevents multiple threads from reading and writing the same values. If you do this properly you can keep track of all the threads.

 

As an extra precaution, I put an if statement when I increment and decrement the thread counter to ensure that it does not exceed the max threads variable before I increment it and above zero before decrement it . These commands are done within a thread safe container. Here is an example of my thread control code: -

 

Remember, you will need the 'Thread Safe Container Plugin' http://www.sendspace.com/file/f0cnbx

ui drop down("Threads", "1,2,3,4,5,6,7,8,9,10,15,20,25,30,35,40,45,50", #uiThreadsMAX)
ui button("Do Some Stuff") {
    if($not(#isRunning)) {
        then {
            set(#isRunning, $true, "Global")
            threadsRun()
        }
        else {
            alert("Please wait for submission to finish!")
        }
    }
}
ui stat monitor("Active Threads: ", #threadsActive)
divider
define _threadsInitialise {
    comment("********** Set thread count **********")
    set(#threadCount, #uiThreadsMAX, "Global")
    comment("********** Reset active thread count **********")
    set(#threadsActive, 0, "Global")
}
define _threadsWait {
    comment("********** Wait for all threads to finish **********")
    comment("********** Give time for threads to fire up **********")
    wait(5)
    loop while($comparison(#threadsActive, ">", 0)) {
        wait(1)
    }
}
define _threadRun {
    plugin command("Open.Framework.dll", "Thread safe container") {
        if($comparison(#threadsActive, "<", #uiThreadsMAX)) {
            then {
                increment(#threadsActive)
                comment("********** Example safe global variable (LOCAL) **********")
                set(#safe, #threadsActive, "Local")
            }
        }
    }
    in new browser {
        comment("********** Fake some code **********")
        navigate("http://google.com/", "Wait")
        type text(<name="q">, "This is a search from thread {#safe}", "Standard")
        comment("********** Fake some delay below **********")
        wait($rand(5, 10))
    }
    divider
    comment("********** Kill new thread and track **********")
    plugin command("Open.Framework.dll", "Thread safe container") {
        if($comparison(#threadsActive, ">", 0)) {
            then {
                decrement(#threadsActive)
            }
        }
    }
}
divider
define threadsRun {
    comment("********** Initialise threads **********")
    _threadsInitialise()
    comment("********** Run threads **********")
    loop(#threadCount) {
        thread {
            _threadRun()
        }
    }
    comment("********** Wait for all threads to finish **********")
    _threadsWait()
    comment("********** Set state to stopped **********")
    set(#isRunning, $false, "Global")
    alert("Submission finished!")
}

The above combination has completely eliminated the problem for me and works like a charm :)

thread control.ubot

  • Like 1
Link to post
Share on other sites

thebigweb example does work i'm doing something similar for my browser bots and low thread http bots.

 

but when you come to multithreading http over a hundred it still has issues. I believe there are two issues here

 

1.variables not being thread safe.

2.variables not updating quick enough and counts being lost

 

as you can see even with the thread safe container go over a certain amount of threads with a lot of cycles and ubot misses some decrements

 

example you will need the plugin linked in previous post

 

you can even drop the thread count down to around 100 and the issue is still there

set(#usedthreads, 0, "Global")
set(#maxthreads, 300, "Global")
loop(6000) {
    loop while($comparison(#usedthreads, ">=", #maxthreads)) {
        wait(1)
    }
    plugin command("Open.Framework.dll", "Thread safe container") {
        increment(#usedthreads)
    }
    thread {
        wait($rand(1, 5))
        plugin command("Open.Framework.dll", "Thread safe container") {
            decrement(#usedthreads)
        }
    }
}


Link to post
Share on other sites

Could this be considered a slow but very stable solution?

 

Limit multithreading to say max 10-15 threads? - (while running, do regular memory cleaning + when the bot starts, set a timer - maybe on load?- this is to shut the bot down after say 20 minutes.-the bot will check if this target has been reached or not - maybe via a separate thread...?)

Also perform  'close page' for 'new browsers' -  at the end of code block.

After 20 minutes - save data + close bot + optionally before shut down ,let the main program 'open' a previously compiled  bot ->which would just clean memory (os addin plugin?)- and restart the main bot and then shut itself down.

If adding delay helps in threads - how much and where should we add them?

 

While that would help overall with the performance of a bot - especially one running long term. It won't help with the threading issue because they die out incredibly fast. You may start with 10 threads and be down to 1 a minute later.

Link to post
Share on other sites

I don't get it?!! How do you guys sell bots if Multi threading is so badly implemented? Who would buy such bots? It doesn't make any sense..  :(

 

I don't mean to offend anyone..  I consider myself a total beginner/learner of Ubot so I am just curious how is it possible to create stable multi threaded apps for sale with so many issues...

 

Thanks.

Link to post
Share on other sites

I don't get it?!! How do you guys sell bots if Multi threading is so badly implemented? Who would buy such bots? It doesn't make any sense..  :(

 

I don't mean to offend anyone..  I consider myself a total beginner/learner of Ubot so I am just curious how is it possible to create stable multi threaded apps for sale with so many issues...

 

Thanks.

 

Well it wasn't really a problem when using the in browser commands since the thread count on average wouldn't be able to go higher than 5-10. But now that we are all using sockets our bots can now handle 100+. threads. Only problem now is what HelloInsomnia stated:

 

 

You may start with 10 threads and be down to 1 a minute later.

 

 

Hopefully UBotDev.com's plugin gets approved soon. I have had the privilege of testing it out and you can easily tell that the global variable problem is limiting the power of threaded bots.

Link to post
Share on other sites

It definitely worked for me. I was having the exact same problem as described. Here is how I used it: -

 

when you increment the global thread count varable put it inside a 'thread safe container'. This will force all other threads to wait until the code within the container has executed. You also need to put reading values and writing values to global shared variables and lists within this container. This prevents multiple threads from reading and writing the same values. If you do this properly you can keep track of all the threads.

 

As an extra precaution, I put an if statement when I increment and decrement the thread counter to ensure that it does not exceed the max threads variable before I increment it and above zero before decrement it . These commands are done within a thread safe container. Here is an example of my thread control code: -

 

Remember, you will need the 'Thread Safe Container Plugin' http://www.sendspace.com/file/f0cnbx

 

I've tested the code you posted above with one change, which is I replaced "navigate" part with 1 second fixed delay and I noticed that that approach is not as safe as you might think.

 

It looks like plugin is writing to disk, which also fails with a lot of thread's executing fast: http://screencast.com/t/Cs2hn2wGERkd

(I get error popups which I didn't get with your code).

 

However, else it looks like all threads were closed successfully (according to this short run).

 

TEST CODE:

ui drop down("Threads", "1,2,3,4,5,6,7,8,9,10,15,20,25,30,35,40,45,50", #uiThreadsMAX)
ui button("Do Some Stuff") {
    if($not(#isRunning)) {
        then {
            set(#isRunning, $true, "Global")
            threadsRun()
        }
        else {
            alert("Please wait for submission to finish!")
        }
    }
}
ui stat monitor("Active Threads: ", #threadsActive)
define threadsRun {
    comment("********** Initialise threads **********")
    _threadsInitialise()
    comment("********** Run threads **********")
    loop(#threadCount) {
        thread {
            _threadRun()
        }
    }
    comment("********** Wait for all threads to finish **********")
    _threadsWait()
    comment("********** Set state to stopped **********")
    set(#isRunning, $false, "Global")
    alert("Submission finished!")
}
divider
define _threadsInitialise {
    comment("********** Set thread count **********")
    set(#threadCount, #uiThreadsMAX, "Global")
    comment("********** Reset active thread count **********")
    set(#threadsActive, 0, "Global")
}
define _threadRun {
    plugin command("Open.Framework.dll", "Thread safe container") {
        if($comparison(#threadsActive, "<", #uiThreadsMAX)) {
            then {
                increment(#threadsActive)
                comment("********** Example safe global variable (LOCAL) **********")
                set(#safe, #threadsActive, "Local")
            }
        }
    }
    in new browser {
        comment("********** Fake some code **********")
        wait(1)
    }
    divider
    comment("********** Kill new thread and track **********")
    plugin command("Open.Framework.dll", "Thread safe container") {
        if($comparison(#threadsActive, ">", 0)) {
            then {
                decrement(#threadsActive)
            }
        }
    }
}
define _threadsWait {
    comment("********** Wait for all threads to finish **********")
    comment("********** Give time for threads to fire up **********")
    wait(5)
    loop while($comparison(#threadsActive, ">", 0)) {
        wait(1)
    }
}
divider

Link to post
Share on other sites

Hopefully UBotDev.com's plugin gets approved soon. I have had the privilege of testing it out and you can easily tell that the global variable problem is limiting the power of threaded bots.

I can imagine that un-reviewed plugin popup is annoying. :)

 

Would love to see this plugin release soon. 

Still waiting for approval. :/

 

Are single threaded applications obsolete? would like to know what the community has to say.

It's not obsolete, it's just that if you have a lot of "tasks" to process you can't wait 1 year (12 months) for single threaded bot to finish (because data is usually needed asap), instead you could use 12 threads/proxies and get the job done in one month, or even use 100 threads/proxies to complete all tasks in just 4 days. Would you prefer to wait one year? :)

 

So my opinion is that single threaded applications can still be used where speed/time is not an important factor...

Link to post
Share on other sites

 

I've tested the code you posted above with one change, which is I replaced "navigate" part with 1 second fixed delay and I noticed that that approach is not as safe as you might think.

 

It looks like plugin is writing to disk, which also fails with a lot of thread's executing fast: http://screencast.com/t/Cs2hn2wGERkd

(I get error popups which I didn't get with your code).

 

However, else it looks like all threads were closed successfully (according to this short run).

 

TEST CODE:

ui drop down("Threads", "1,2,3,4,5,6,7,8,9,10,15,20,25,30,35,40,45,50", #uiThreadsMAX)
ui button("Do Some Stuff") {
    if($not(#isRunning)) {
        then {
            set(#isRunning, $true, "Global")
            threadsRun()
        }
        else {
            alert("Please wait for submission to finish!")
        }
    }
}
ui stat monitor("Active Threads: ", #threadsActive)
define threadsRun {
    comment("********** Initialise threads **********")
    _threadsInitialise()
    comment("********** Run threads **********")
    loop(#threadCount) {
        thread {
            _threadRun()
        }
    }
    comment("********** Wait for all threads to finish **********")
    _threadsWait()
    comment("********** Set state to stopped **********")
    set(#isRunning, $false, "Global")
    alert("Submission finished!")
}
divider
define _threadsInitialise {
    comment("********** Set thread count **********")
    set(#threadCount, #uiThreadsMAX, "Global")
    comment("********** Reset active thread count **********")
    set(#threadsActive, 0, "Global")
}
define _threadRun {
    plugin command("Open.Framework.dll", "Thread safe container") {
        if($comparison(#threadsActive, "<", #uiThreadsMAX)) {
            then {
                increment(#threadsActive)
                comment("********** Example safe global variable (LOCAL) **********")
                set(#safe, #threadsActive, "Local")
            }
        }
    }
    in new browser {
        comment("********** Fake some code **********")
        wait(1)
    }
    divider
    comment("********** Kill new thread and track **********")
    plugin command("Open.Framework.dll", "Thread safe container") {
        if($comparison(#threadsActive, ">", 0)) {
            then {
                decrement(#threadsActive)
            }
        }
    }
}
define _threadsWait {
    comment("********** Wait for all threads to finish **********")
    comment("********** Give time for threads to fire up **********")
    wait(5)
    loop while($comparison(#threadsActive, ">", 0)) {
        wait(1)
    }
}
divider

 

Strange, I tried the modified code above but didn't receive any errors.

Link to post
Share on other sites
  • 3 weeks later...

No plugin key yet...I'm not sure what's taking so long...it's over a month and still no key. A few days ago they asked me if I still need the key plugin, and I said yes, else I wouldn't request it, but still no key till today. Not sure what is going on there but it looks like support is getting worse and worse.

Link to post
Share on other sites

Hello!

I am assuming this is regarding ticket JAT-686-55550, where two free plugin keys and a paid plugin key were requested. 

The day after this request was made, I asked the requester to narrow down which free plugin key he wanted first, and did not hear back. I sent an additional response to remind the requester that he had not responded. 

If you have responded, there may an issue with the email or updating system on your ticket? Those responses did not come through. I've been hoping for a clarification!


Jason

 

PS - Feel free to message me here so we can see what the problem is.

Link to post
Share on other sites

It's all OK guys, I've got the plugin key and I didn't have to pay for this one... I have to pay for one that I'm thinking to sell.

 

I'm going to release "threads counter" plugin today or tomorrow, just need to get some things ready first.

  • Like 1
Link to post
Share on other sites

Well it wasn't really a problem when using the in browser commands since the thread count on average wouldn't be able to go higher than 5-10. But now that we are all using sockets our bots can now handle 100+. threads.

 

 

Gogetta : Did you used socket on v.4 or v5 ? If in v.4 did you meant socket just like seth explain on ubot video (the one that disable images, css, javascript,etc) ?

 

I ask because i never found how to use real socket on v.4. But if you meant was v.5 then this thread is for v.4 or do i miss something?

 

It's all OK guys, I've got the plugin key and I didn't have to pay for this one... I have to pay for one that I'm thinking to sell.

 

Where can we buy your paid plugin?

Link to post
Share on other sites

Where can we buy your paid plugin?

I'm not selling the paid one yet if you mean that one (comming soon), but if you mean the "Threads Counter" plugin, that one is free.

 

Else I think Gogetta was using Aymens sockets plugin...

  • Like 1
Link to post
Share on other sites

I'm not selling the paid one yet if you mean that one (comming soon), but if you mean the "Threads Counter" plugin, that one is free.

 

Else I think Gogetta was using Aymens sockets plugin...

 

Thanks UbotDev

Link to post
Share on other sites

I tried to contact the support about this issue but they rejected the bug: http://tracker.ubotstudio.com/issues/208

 

Lilly actually suggested that sufficient delay should solve that problem (to bad we didn't think of that lol) and rejected the bug without me having any chance to explain.

 

So much about the support....

Link to post
Share on other sites

I tried to contact the support about this issue but they rejected the bug: http://tracker.ubotstudio.com/issues/208

 

Lilly actually suggested that sufficient delay should solve that problem (to bad we didn't think of that lol) and rejected the bug without me having any chance to explain.

 

So much about the support....

 

Part of what we do at support is looking through the script you presented in your ticket and determining that your issue was not a bug. If you run the script and you find that the logic you created is failing after our amended fix, then feel free to send it back and we will run it again. Keep in mind that I ran you script many times after implementing the change and it never failed at any point during the testing. 

 

Here is a copy of the amended script you presented with your issue. After extensive dissection, a delay in the proper place will show you that the threads are closing without issue. There were no issues noticed in the decrement or increment command. The delay is 0.2 seconds, and the suggestion is to allow a little time in between each thread before opening a new one. 0.2 seconds will do just fine.

Here is the script in case anyone else finds it helpful:

 

ui drop down("COUNT", "100,1000,10000", #Input Count)

ui drop down("Threads", "10,20,30,40,50", #INPUT Threads)

ui stat monitor("Threads (Active/Total):", "{#THREADS Active}/{#THREADS Max}")

ui stat monitor("COUNT:", #COUNT)

set(#COUNT, #Input Count, "Global")

set(#THREADS Max, #INPUT Threads, "Global")

set(#THREADS Active, 0, "Global")

loop while($comparison(#COUNT, ">", 0)) {

if($comparison(#THREADS Active, "

then {

increment(#THREADS Active)

THREAD START()

decrement(#COUNT)

wait(0.2)

}

else {

wait(0.2)

}

}

}

set(#WAIT Count, 0, "Global")

loop while($both($comparison(#THREADS Active, ">", 0), $comparison(#WAIT Count, "

increment(#WAIT Count)

wait(2)

}

if($comparison(#THREADS Active, ">", 0)) {

then {

alert("Some threads were not closed/decremented properly.")

}

else {

alert("All threads were closed")

}

}

define THREAD START {

thread {

wait(3)

decrement(#THREADS Active)

}

}

 

 

If anyone following this thread finds an issue with this threading example after running it, simply let us know and we'll be happy to take a look at support.

 

Thank you everyone!

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...