The 2 things you would change in Ubot Studio if you could?

[Seth Turin 223]

This thread should be closed now, its lost the spirit of Dan's original question now, and is becoming a vacuum for complaints. 

Ah, Thank you!

 

Quickly though, I'd like to try and explain what I think you guys are seeing with threading. Imagine a fruit bowl filled with fruit, and a bunch of hungry gorillas. In normal program flow, the gorillas stand in line, each awaiting his turn to take a piece of fruit from the bowl. If you have a zookeeper keeping count of how much fruit there is, he will have no problem doing so. Now imagine the same scenario, but instead of lining up, the gorillas may take fruit whenever they please in a free-for-all. Two gorillas might take fruit at the same time, or several seconds might pass before any fruit is taken, or any other number of unusual fruit counting might happen. To the zookeeper keeping count, it is no longer a clean and simple job. This is multithreading.

 

If that zookeeper is also adding fruit, it also becomes an issue of how much fruit he adds at a time, whether he does so synchronously or asynchronously. When the zookeeper looks at his tallies at the end of the day, unless he counted very carefully, the numbers might not look quite right. Looking at the situation, it's less likely that the gorillas or fruits are magically disappearing, and more likely that the counting is just a crazy process when it comes to threading. 

 

In 100% of the threading issues I've looked at, I've been able to solve it by moving where the increments/decrements are placed.

[kev123 132]

Ah, Thank you!

 

Quickly though, I'd like to try and explain what I think you guys are seeing with threading. Imagine a fruit bowl filled with fruit, and a bunch of hungry gorillas. In normal program flow, the gorillas stand in line, each awaiting his turn to take a piece of fruit from the bowl. If you have a zookeeper keeping count of how much fruit there is, he will have no problem doing so. Now imagine the same scenario, but instead of lining up, the gorillas may take fruit whenever they please in a free-for-all. Two gorillas might take fruit at the same time, or several seconds might pass before any fruit is taken, or any other number of unusual fruit counting might happen. To the zookeeper keeping count, it is no longer a clean and simple job. This is multithreading.

 

If that zookeeper is also adding fruit, it also becomes an issue of how much fruit he adds at a time, whether he does so synchronously or asynchronously. When the zookeeper looks at his tallies at the end of the day, unless he counted very carefully, the numbers might not look quite right. Looking at the situation, it's less likely that the gorillas or fruits are magically disappearing, and more likely that the counting is just a crazy process when it comes to threading. 

 

In 100% of the threading issues I've looked at, I've been able to solve it by moving where the increments/decrements are placed.

how about correcting one of the examples of  it not working or do a quick multithread example. It would put the issue to bed

[Bot-Factory 602]

Yes please...Seth Turin.. here is the challenge:

 

 

Goal:

90-100 Threads in parallel. So please no additional waits.

And the list should have exactly 10000 entries at the end. 

 

if you can change this code so that it works in V4 and V5 without any crashes, no threads going down (constantly running 100threads until the end) and exactly 10000 entries in the list, then I will never say a bad word about threading again..

 

Oh I forgot.. No UI / bot freezes. Constantly updating the stat mons. And no 500+MB memory usage. 

 

Because all of this is working when we use Plugins. But hopefully we are just to stupid to do it right with the native commands. 

I really would love to make this working without plugins. 

 

So please.. show us the light!

 

ui stat monitor("Open Threads:", #threadcount)
ui stat monitor("Total:", $list total(%numbers))
ui button("Stop") {
    set(#threadcount, 0, "Global")
    clear list(%numbers)
}

ui button("Start High (100Threads / 5000Loop)") {
    set(#Loops, 10000, "Global")
    set(#threads, 100, "Global")
    RunTest1()
    alert("Done")
}
define RunTest1 {
    clear list(%numbers)
    set(#threadcount, 0, "Global")
    loop(#Loops) {
        loop while($comparison(#threadcount, ">=", #threads)) {
            wait(0.1)
        }
        increment(#threadcount)
        RunTest1thread($random text(50))
    }
    loop while($comparison(#threadcount, ">", 0)) {
        wait(0.1)
    }
}
define RunTest1thread(#number) {
    thread {
    wait(0.2)
        add item to list(%numbers, #number, "Don\'t Delete", "Global")
        decrement(#threadcount)
    }
}

 

Test file:

https://dl.dropboxusercontent.com/u/10322/Threads%20and%20Add%20Items%20%28test%29.ubot

It has all my examples. And it will also show examples how it works with plugins without any problems. 

 

Dan

[Code Docta (Nick C.) 638]

Personally I think that Ubot can not do this yet at all.

 

1. Gorilla's need a ticket dispenser (aka thread ID)

2. Zookeeper needs to call ticket number when previous Gorilla's are served

3. Gorilla's come get food in orderly fashion

 

Otherwise, you are just throwing food at the Gorilla's, missing some and others get too much fruit in all of the wrong places.

 

So, I agree with Kev123 and he has been throwing the thread ID idea around a bit too. But, yes if you have tested this it should nothing for you to post the code here or in a thread of it's own. So, if you have solved the issue then why the analogy?

 

 

Inside each thread container we should be able to see what is going on in each thread by thread ID EASILY, but not the case in Ubot.

 

So, Please do show an example this would be my first thing I would like to see.

 

The second is, real thread control because I think we are just throwing fruit at the Gorillas ..like ehm never mind.

 

TC

[Bot-Factory 602]

         

[MiriamMB 63]

Hello Everyone,

 

Please keep your responses pertinent to the topic of this thread.

 

The topic has already veered far off topic, but we understand that it is an important conversation.

 

All we ask is that you adhere to the forum rules and refrain from rudeness.

 

The forum rules still apply and this thread, along with all threads on the forum will be moderated. Rude discourteous posts that are not relevant to the topic will be remove. 

 

If you have any thoughts you'd like to direct to admin about how to run the forum, thoughts to UBot staff members, or thoughts to other users, you can send it via PM or use the support portal as advised in the forum rules.

 

Thank you. 

[the_way 52]

When ubot starts up i want it to say in a hot computer girl voice, (like the girl guide in halo)

 

Hello again,<my name> how are you?

 

i can then speak into my microphone, and say, "i'm good darling, i like your new earrings"

 

It then recognises me, and says "quit the flattery, and get back to work"

[the_way 52]

Hey! i made a thread for posting a ubot tip 

http://www.ubotstudio.com/forum/index.php?/topic/16337-post-a-quick-ubot-tip/

 

and only a few people posted.

 

Yet if you looked at this thread, you would think that most people spend their time mostly complaining, instead of posting good ideas

[Seth Turin 223]

Yes please...Seth Turin.. here is the challenge:

 

 

Goal:

90-100 Threads in parallel. So please no additional waits.

And the list should have exactly 10000 entries at the end. 

 

if you can change this code so that it works in V4 and V5 without any crashes, no threads going down (constantly running 100threads until the end) and exactly 10000 entries in the list, then I will never say a bad word about threading again..

 

Oh I forgot.. No UI / bot freezes. Constantly updating the stat mons. And no 500+MB memory usage. 

 

Because all of this is working when we use Plugins. But hopefully we are just to stupid to do it right with the native commands. 

I really would love to make this working without plugins. 

 

So please.. show us the light!

 

ui stat monitor("Open Threads:", #threadcount)

ui stat monitor("Total:", $list total(%numbers))

ui button("Stop") {

    set(#threadcount, 0, "Global")

    clear list(%numbers)

}

 

ui button("Start High (100Threads / 5000Loop)") {

    set(#Loops, 10000, "Global")

    set(#threads, 100, "Global")

    RunTest1()

    alert("Done")

}

define RunTest1 {

    clear list(%numbers)

    set(#threadcount, 0, "Global")

    loop(#Loops) {

        loop while($comparison(#threadcount, ">=", #threads)) {

            wait(0.1)

        }

        increment(#threadcount)

        RunTest1thread($random text(50))

    }

    loop while($comparison(#threadcount, ">", 0)) {

        wait(0.1)

    }

}

define RunTest1thread(#number) {

    thread {

    wait(0.2)

        add item to list(%numbers, #number, "Don\'t Delete", "Global")

        decrement(#threadcount)

    }

}

 

Test file:

https://dl.dropboxusercontent.com/u/10322/Threads%20and%20Add%20Items%20%28test%29.ubot

It has all my examples. And it will also show examples how it works with plugins without any problems. 

 

Dan

Ok, I spent a little time playing with this. This one is  different than any of the examples I had seen so far. When I run the script, it does actually crash. When I drilled down into it, I can confirm that this is genuinely a thread safety issue.

 

Now, I still want to draw a distinction between this and counting issues. Kev was correct when he said:

 

What also makes the term "thread safe" on the forum more confusing is people are actually talking about two different scenarios and packaging it up in one term.

The counting issues are just an artifact of multithreading being genuinely difficult to do correctly. The crashing issue is because lists are not thread safe. And actually, neither are variables. 

 

This is actually kind of a sticky problem to fix, and rather than spend a long time fixing a broken command that is difficult to use in the first place, I'd like to, instead, work on a better threading system from the ground up. The whole point of UBot is to simplify complex ideas, and I have some ideas for how we might do that with threading. In the mean time, if you need to use threading, I suggest doing it in UBot 4.

 

To anyone reading this thread who isn't a UBot veteran: these guys have all been using UBot for years. Multithreading is one of the most advanced things you can do with UBot, and one of the most difficult. I'd recommend waiting until you're comfortable with all the other commands before messing with threading.

[Bot-Factory 602]

Thank you Seth for the open words! Really appreciate that!

[LoWrIdErTJ - BotGuru 904]

Ok, I spent a little time playing with this. This one is  different than any of the examples I had seen so far. When I run the script, it does actually crash. When I drilled down into it, I can confirm that this is genuinely a thread safety issue.

 

Now, I still want to draw a distinction between this and counting issues. Kev was correct when he said:

 

The counting issues are just an artifact of multithreading being genuinely difficult to do correctly. The crashing issue is because lists are not thread safe. And actually, neither are variables. 

 

This is actually kind of a sticky problem to fix, and rather than spend a long time fixing a broken command that is difficult to use in the first place, I'd like to, instead, work on a better threading system from the ground up. The whole point of UBot is to simplify complex ideas, and I have some ideas for how we might do that with threading. In the mean time, if you need to use threading, I suggest doing it in UBot 4.

 

To anyone reading this thread who isn't a UBot veteran: these guys have all been using UBot for years. Multithreading is one of the most advanced things you can do with UBot, and one of the most difficult. I'd recommend waiting until you're comfortable with all the other commands before messing with threading.

 

 

It isnt all that hard.

In v4 all we had to do is turn on debug mode while in ubot with our code and ran correctly.  Where as compiled dont have that option to do so, and the debug mode logs everything to file.

We just need that enabled without logging to file and it would solve the issue.

 

Tested between myself and Eddie back with v4.  and assume it is still the same in v5 (feel free to test v5 users)

[Bot-Factory 602]

It isnt all that hard.

In v4 all we had to do is turn on debug mode while in ubot with our code and ran correctly.  Where as compiled dont have that option to do so, and the debug mode logs everything to file.

We just need that enabled without logging to file and it would solve the issue.

 

Tested between myself and Eddie back with v4.  and assume it is still the same in v5 (feel free to test v5 users)

Not sure if I understand you correctly. Enabling debug mode in ubot studio? And that should fix what? The crash in v5?

Or the issue with global variables and global list.

 

In v4 the fix is easy.I use two plugins :-) But in v5 it's crashing completely. Also plugins don't help here.

Dan

[Marton 48]

The counting issues are just an artifact of multithreading being genuinely difficult to do correctly.

 

Thank you for getting back to us on this! However, the counting issue DOES cause problems on properly coded multi-threaded applications as well. As long as there is a variable inside the thread which is incremented/decremented, a miscount could happen, it's just a matter of when. I opened a thread which only addresses the counting issue, would you mind taking a look at the numbers?

 

http://www.ubotstudio.com/forum/index.php?/topic/16416-simple-proof-variables-are-not-thread-safe-in-ubot-45/ (don't mind the title please, I'm only talking about the counting issue here)

 

My thread is NOT about the best way to code multi-threaded bots. This just shows that if two (or more) threads finish at the same time (or very close to each other) the variable which is counting the threads (or in fact any other variable) won't be incremented/decremented properly (and you never know when this kind of thing could happen!). And this can happen even with delays, it's just much less likely (and with the best algorithms too), I really hope you get that.

 

But there is a solution to all this! For now the solution is to use UbotDev's (or Pash's) plugin (see the case study in my thread). It's quite a simple solution, and I hope you'll have a minute to take a look at it (by the way I sent my open letter to you through PM, it is about this very same issue).

 

And unfortunately the issue is present in Ubot 4 too (but at least it doesn't crash).

 

I'm really sorry that we are bothering you with this issue all the time, but I'm only doing it so you'll take this into account too when you're making the planned threading changes in Ubot in the future.

 

Thanks and let me know if you have any questions, I can show you some better codes to prove how the counting issue actually affects some properly coded multi-threaded applications.

 

Marton

[kev123 132]

It isnt all that hard.

In v4 all we had to do is turn on debug mode while in ubot with our code and ran correctly.  Where as compiled dont have that option to do so, and the debug mode logs everything to file.

We just need that enabled without logging to file and it would solve the issue.

 

Tested between myself and Eddie back with v4.  and assume it is still the same in v5 (feel free to test v5 users)

correctly me if i'm wrong but debug likely just reduces the chance or adds a delay to access to the variable I think it would be just be a work around.

[kev123 132]

Ok, I spent a little time playing with this. This one is  different than any of the examples I had seen so far. When I run the script, it does actually crash. When I drilled down into it, I can confirm that this is genuinely a thread safety issue.

 

Now, I still want to draw a distinction between this and counting issues. Kev was correct when he said:

 

The counting issues are just an artifact of multithreading being genuinely difficult to do correctly. The crashing issue is because lists are not thread safe. And actually, neither are variables. 

 

This is actually kind of a sticky problem to fix, and rather than spend a long time fixing a broken command that is difficult to use in the first place, I'd like to, instead, work on a better threading system from the ground up. The whole point of UBot is to simplify complex ideas, and I have some ideas for how we might do that with threading. In the mean time, if you need to use threading, I suggest doing it in UBot 4.

 

To anyone reading this thread who isn't a UBot veteran: these guys have all been using UBot for years. Multithreading is one of the most advanced things you can do with UBot, and one of the most difficult. I'd recommend waiting until you're comfortable with all the other commands before messing with threading.

Agree it would be madness to start chucking locks around everything generally can cause more issues than it solves(done that with the latest plugin I created to prove concept but not a long term sol its a mash of code so i'll be modifying soon.

 

I've got a pretty sweet well known threading dll that I use. Created a non verified plugin to test it out needs a lot of work around's due to limits of api but works very well in ubot. Its like beast mode for http/socket guys the main reason I never created a plugin was due to not being able to access what ever you do in threads to ensure browser is closed. I think it would work well with ubot core if added.

[Seth Turin 223]

Yeah, there's no need to argue the counting issue. In either case, thread safe is an issue and we need a better parallel processing system. So getting back to the spirit of this thread, here are my two:

 

1. Simplified threading system, to make thread safety a non-issue and to make thread complexity easier to manage

 

2. Object oriented design capabilities, to help organize large bots, but without overcomplicating the basic scripting system for beginners.

[Code Docta (Nick C.) 638]

+1 Seth, Thank you for some action on this.

[a2mateit 395]

This is the #1 thing I would change/implement!:

http://www.ubotstudio.com/forum/index.php?/topic/12703-the-ultimate-feature-suggestion/

[wtf 4]

+1 Seth, Thank you for some action on this.

+1 if we actually get action and not just words

[the_way 52]

OHHH!!! i have a great one!, 

 

special functionality to allow ubot .exes to talk to each other by some sort of external_variable they pass each other.

[kev123 132]

OHHH!!! i have a great one!, 

 

special functionality to allow ubot .exes to talk to each other by some sort of external_variable they pass each other.

sounds a good idea I have considered it for a plugin but talking between remote computers as well. My only concern was getting security right and not making it to easy for bad types to do certain things.

[Sanjeev 46]

Kev,  Is there any windows utility  or any other means by which we can get 2 bots to communicate with each other currently?

Thanks

[Bot-Factory 602]

Ok, I spent a little time playing with this. This one is  different than any of the examples I had seen so far. When I run the script, it does actually crash. When I drilled down into it, I can confirm that this is genuinely a thread safety issue.

 

Now, I still want to draw a distinction between this and counting issues. Kev was correct when he said:

 

The counting issues are just an artifact of multithreading being genuinely difficult to do correctly. The crashing issue is because lists are not thread safe. And actually, neither are variables. 

 

This is actually kind of a sticky problem to fix, and rather than spend a long time fixing a broken command that is difficult to use in the first place, I'd like to, instead, work on a better threading system from the ground up. The whole point of UBot is to simplify complex ideas, and I have some ideas for how we might do that with threading. In the mean time, if you need to use threading, I suggest doing it in UBot 4.

 

To anyone reading this thread who isn't a UBot veteran: these guys have all been using UBot for years. Multithreading is one of the most advanced things you can do with UBot, and one of the most difficult. I'd recommend waiting until you're comfortable with all the other commands before messing with threading.

 

Seth,

 

can you say something about the timeframe for the threading "fix" / new threading function. Just a hint if this is something we can expect in V5 within the next 1-3 months? 

Or do we talk about something for v6 in 2015?

 

Dan

[Brutal 164]

1- I spend a fortune on addons from other developers here in the buy/sell thread.... These are things that just make ubot more functional, so the primary thing I want is to be getting those things for my monthly fees rather than mostly nothing.

 

Don't get me wrong - I love that those talented individuals are creating a better ubot environment, but it just seems fair that those of us who pay monthly without fail would be getting those things from ubot staff as updates/enhancements.... Let NON monthly payers pay the guys for individual addons in the buy/sell thread.

 

So, when people apply to sell addons, make it a part of the user agreement that they pass you the code, and you make that a standard part of what monthly clients get as they come out - and again, let the non monthly buy them on a per piece basis out of the buy/sell thread. - And if a person opts-out of monthly payments, disable all "non-standard" addons and let them go buy them out of the buy/sell thread.

 

2- There is no "2" ... with the functionality and ui addons that people have made here, there just isn't anything more I could hope for.

[UBOTEM 18]

Yeah, there's no need to argue the counting issue. In either case, thread safe is an issue and we need a better parallel processing system. So getting back to the spirit of this thread, here are my two:

 

1. Simplified threading system, to make thread safety a non-issue and to make thread complexity easier to manage

 

2. Object oriented design capabilities, to help organize large bots, but without overcomplicating the basic scripting system for beginners.

 

 

Nice to see active on this thread

 

My thoughts on your two points...

 

1. I think this is already done with SmartThreads, and works without fail on UB4.

Just needs to be implemented in core of UB5 I see the spawn thread was supposed to do this but it is a failure as no way to have a thread ID, and nothing seems to solve this in my testing...

 

2. This seems interesting, and I always get a little stuck looking at my code a year later, if there was better structure to code this could be cool.

 

 

But I think most of the core issues are fixed and working in UB4 thanks to others making cool plugins to patch up the few bugs that was left in UB4.

 

UB5 should be robust!!

But I feel it is not

 

Its faster to work with and cool for some single thread tools, but if I want to make a full bot with multi thread then I have to used UB4.

 

Also why is large data tables and lists still an issue? this should have been default!!

 

So I guess this leaves me to my two...

 

My Suggestion of two things to change on UB5 are.....

 

** 1 **

ThreadSpawn command should work with a default thread id like in the SmartThread plugin for UB4. Then can control the thread and be safe.

 

** 2 **

Large Data Tables & Large Lists should be fine in UB5 (solved in UB4 with LargeData Plugin)