[Sell] Smart Thread Plugin - Easy threading

[kev123 132]

pm sent

[UBOTEM 18]

Just bought

 

Have to sleep, but will try out tomorrow

 

Hopefully it may solve an issue I currently have with HTTP Get activated is stopping my threads to fire to max value.

 

I was using the free thread counter and script from ubotdev but seems to have issue when switch to http get instead of browse and use $documentText.

 

Anyway will try out and report if fixed issue, as may solve for others two

 

Looks simple easy to setup GREAT JOB

[UBOTEM 18]

 

[FREE] Scrape Framework

 

Here is a free scraping framework for you guys. It's using

the smartthread plugin and the large data plugin from Kev.

 

DOWNLOAD

 

Hello Dan,

 

Link does not work anymore?

Can you send me a PM or something please

[UBOTEM 18]

Umm... maybe I did something wrong? but has slowed down my bot!

 

Stripped my bot of old thread code.

 

moved thread process into a define without thread.

 

then placed

plugin command("Smartthreads.dll", "Thread control normal", "THREAD START", #COUNT, #THREADS Max, "No")

where my old multi thread loop was.

 

tested and working fine, and shows all threads are being used.

but seemed slower! (when was hoping for faster)

 

not changed anything else just swapped over to your smart threads.

 

So I opened both versions of my compiled bot, new one with smart threads, and old previous.

 

Set both up to do same job, with only 2 threads, started yours 2 min before my old multi-threaded version, and about 5 min in my old one took over!

 

The main issue with my previous version of my bot, that was hoping would fix issue (which sort of has)

 

I have a switch to change scrape method like:

if($comparison(#input_method, "=", "HTTP Get")) {
    then {
        set(#http_get, $plugin function("HTTP post.dll", "$http get", #var_url, #var_userAgent, #var_referer, "", "{#var_httpGetTimeout}000"), "Local")
    }
    else {
        in new browser {
            set user agent(#var_userAgent)
            set referrer(#var_referer)
            navigate(#var_url, "Wait")
            wait for browser event("Everything Loaded", #var_httpGetTimeout)
            set(#http_get, $document text, "Local")
            wait(1)
            close page
        }
    }
}

And when I switch to HTTP it never fired threads above 13, yet browser mode always max-ed to whatever set.

 

Your plugin has defiantly fixed that as can see that the threads are being used, but seems to be slower?

 

my define for thread is quite simple.

Grabs html using either HTTP or BROWSER as above.

runs quite a few xpath and regex on the html, about 20 checks at moment.

saves to a normal ubot table. (know large tables will help and is next on my list to do, after this issue)

Love your plugin though, very simple, maybe i'm doing something wrong?

 

My plan is to have HTTP threads upto 200 (currently seems very slow at 10 with your plugin, and faster without but never goes above 13ish)

 

Am quite competent, but maybe missing something out?

 

Not tried the proxies yet, but that looks interesting also

Edited March 20, 2015 by UBOTEM

[UBOTEM 18]

BTW switching code over was quite straight forward, and a lot less code

 

Just need to find out why it has slowed down?

[Bot-Factory 602]

BTW switching code over was quite straight forward, and a lot less code

 

Just need to find out why it has slowed down?

 

Well the question is.. did it really slow down?

Not running the same amount of threads in parallel doesn't automatically mean it is slower now.

 

The http request takes a lot less time compared to a browser command. So the threads will finish in a lot less time. 

And that could be the reason why the thread trashhold can not be reached. 

 

Just add a 

wait(1)

to the end of the thread define. And you will see that the thread total will increase. But increasing the threads isn't the goal right.

Completing everything as fast as possible.

 

And if your threads finish very quickly, it's a matter of your PC power on how many threads it can start. 

 

Dan

[UBOTEM 18]

Well the question is.. did it really slow down?

Not running the same amount of threads in parallel doesn't automatically mean it is slower now.

 

The http request takes a lot less time compared to a browser command. So the threads will finish in a lot less time. 

And that could be the reason why the thread trashhold can not be reached. 

 

Just add a 

wait(1)

to the end of the thread define. And you will see that the thread total will increase. But increasing the threads isn't the goal right.

Completing everything as fast as possible.

 

And if your threads finish very quickly, it's a matter of your PC power on how many threads it can start. 

 

Dan

 

Hi Dan,

 

Thanks for help, maybe I am not understanding or wires crossed :/

 

But before when I used the old threading method and http threads never maxed out (so that makes sense that they finished quicker, but still expect the threads to fire out to 20+ or even 200+ and crash!)

I'm sure I tried to add a wait at that point and still never maxed out. (will try a 10 sec wait and see if makes any difference)

 

Anyway moving on.... I switched to smartthreads and now it does fire to the max, but seems to freeze my lappy (running quad core 8gb ram).

 

My test was.... old version vs smartthread version on only 2 http threads each, and old version seemed/was to be faster at process of 500 items.

 

Its really confused me, as in old threaded version I would still expect it to max out!

 

Confused me as does not seem logical

 

 

As for parallel threads, surely both multithread is parallel?

 

Also Dan, can you show us the free framework you shared with smartthreads & big lists?

 

I need to be able to use http with smarthreads and big lists

 

my target is upto 200 threads, does this seem reasonable?

[MMD 7]

Can i run two "Thread control normal" command at the same time without problems? Havent tested it but i hope its not problem.

[kev123 132]

ubotem

 

Have you got a code sample?

 

MMD

 

no because you would need to use ubot internal threads to accomplish this and that could cause problems.

You can however use advanced add commands.

[Bot-Factory 602]

Hi Dan,

 

Thanks for help, maybe I am not understanding or wires crossed :/

 

But before when I used the old threading method and http threads never maxed out (so that makes sense that they finished quicker, but still expect the threads to fire out to 20+ or even 200+ and crash!)

I'm sure I tried to add a wait at that point and still never maxed out. (will try a 10 sec wait and see if makes any difference)

 

Anyway moving on.... I switched to smartthreads and now it does fire to the max, but seems to freeze my lappy (running quad core 8gb ram).

 

My test was.... old version vs smartthread version on only 2 http threads each, and old version seemed/was to be faster at process of 500 items.

 

Its really confused me, as in old threaded version I would still expect it to max out!

 

Confused me as does not seem logical

 

 

As for parallel threads, surely both multithread is parallel?

 

Also Dan, can you show us the free framework you shared with smartthreads & big lists?

 

I need to be able to use http with smarthreads and big lists

 

my target is upto 200 threads, does this seem reasonable?

 

If you want to hear my honets oppinion... Threading with a high number of threads will give you a lot of strange errors with Ubot studio.

But not because of the smartthread plugin or anything else. The main reason is the Ubot Studio API. There are a couple of known "bugs" 

Just look into the tracker and view the API bug reports.

 

I would not waste a lot of time with optimization. There is no way for us to optimize that behavior from the outside. 

I used an ubot scraper to scrape 5 million urls. But instead of optimizing everything. I just ran 6 bots on 3 VPS servers. 

And at the end, I just combined the output of all 6 bots and removed duplicates. Job done. 

 

Scale horizontal (more bots on more servers)

And not vertical (more threads on single bot and single server)

 

Just my 2 cents

Dan

[UBOTEM 18]

OK quick question....

 

Say I detect that a proxy is bad within my thread/define, how do I switch proxy?

 

Or do we have to mark thread bad, to be redone?

[UBOTEM 18]

@dan thanks for advice

 

Also would like to know more about VPS and how to run on there! (also would this mean my Mac customers could use?)

[Bot-Factory 602]

@dan thanks for advice

 

Also would like to know more about VPS and how to run on there! (also would this mean my Mac customers could use?)

I use

vultr.com

 

You can get a windows based server from them to run all your bots. And yes, you can connect to that server from a mac (via RDP) as well.

But that's probably not something for your customers. Because you would have to provide them access to your server. I mean, it's possible. But that's a configuration and scalability issue. 

Depending on what your bot is doing and how many users you are looking for. 

 

if this is just something you sell for a high price to 10-30 users, a terminal server based solution could be a possibility. But definately nothing for a larger scale scenario. 

 

In such a case I would look into developing a web based frontend for your customers. And let that frontend connect to your bot on the backend server. The communication plugin offers some features to get you started.

If that is the kind of scenario you are looking for.

 

Cheers

Dan

[UBOTEM 18]

@dan thanks for info, not quite ready yet for it but will be aiming for a website front-end, and a server hosting bots. User requests job, job gets queued, email user with link to data

Edited March 26, 2015 by UBOTEM

[Hightower 2]

smartthreads plugin is not able to perform required task. I gave it list of 40 list items but it does not perform action on each. it is missing some the list items...  help please

[Bot-Factory 602]

smartthreads plugin is not able to perform required task. I gave it list of 40 list items but it does not perform action on each. it is missing some the list items...  help please

 

Please provide some example code and more details. It's quite hard to understand where you run into problems that way.

I have smart thread plugin running in a couple of scenarios without problems.

 

But you always have to be careful with ubot and threading. There are some limitations.

If threads execute to quickly, or are started and finished within milliseconds, you will always see strange behavior. But that is not related to smartthread plugin. That's just Ubot...

You would see the same behavior with normal threading.

 

Also if you use other plugin commands within a thread, it could cause issues. 

 

I also see it very often that people apply "old" (V4) concepts to ubot studio and the latest updates. 

Like using http container with http plugin. Or using local variable plugin for local variables. 

 

And a lot of that stuff is from the v4 days and don't work in a similar way with v5. 

 

So there is really a lot of stuff that can go wrong here. And without knowing more details, it's not possible to help you.

 

Dan

[kev123 132]

as dan said code example if you can thanks

[UBOTEM 18]

Ahh help, having issues where only 1 set of threads firing out?!

 

Loops @ 6

Threads @ 4

plugin command("Smartthreads.dll", "Thread control normal", "main_prog", #thread_loops, #input_maxThreads, "No")

4 threads fire up!
Also placed an alert at the end of the command 'main_prog' so can see they hit that, but next two do not run, and my bot hangs

 

Using 1.3 I think

 

 

Any help v much appreciated

[UBOTEM 18]

Ahh help, having issues where only 1 set of threads firing out?!

 

Loops @ 6

Threads @ 4

plugin command("Smartthreads.dll", "Thread control normal", "main_prog", #thread_loops, #input_maxThreads, "No")

4 threads fire up!

Also placed an alert at the end of the command 'main_prog' so can see they hit that, but next two do not run, and my bot hangs

 

Using 1.3 I think

 

 

Any help v much appreciated

 

Solved myself

 

Thought was going mad.... So checked threads in use, and realised that I used a normal thread on startup.

Removed this and smartthreads is now doing its job!

[BeerNut 25]

I have just incorporated this plugin into one of my more simple bots. But I am having a strange error where pressing the uBot stop button does not stop the bot anymore. The program will continue to go though the list and keeping going until it finishes the task.

 

Its basically loading a list, then looping the list total and then doing a thing. I am not passing the list to the smart thread system just yet. So inside the thread its pulling next list item. But I am passing the loop (list total) directly to the smart thread.  So it knows it needs to loop 1600 times and will continue to loop 1600 times even if I press the uBot stop button. 

 

It looks like I need to make a new UI to use this plugin and only use your stop and start buttons? Or is this just a bug?

 

Edit: I should mention that I am using Thread control Advanced add, Loop Number, list total (%list) as the loop number, and no proxies. Using uBot 4.

 

Edit 2: I watched the videos and make a new thread using the Thread control Advanced add, List, %list, no proxies. Once again if I press uBot stop the program continues though the list until it finishes. I also tested in uBot 5 and it has the same issue.

[kev123 132]

hi you are correct you need to use stop and start button, there is a command to stop the plugin.

 

thanks

Kev123

[SaHiN 14]

Hi kev,

 

working aymen http post command multi thread?

[kev123 132]

hey sorry for delay in response yes it will work fine if you have problems post here and i'll take a look,

[SaHiN 14]

hey sorry for delay in response yes it will work fine if you have problems post here and i'll take a look,

 

please sample code?

[kev123 132]

comment("add some sites to list just for example")
clear list(%urls)
add item to list(%urls,"http://www.ubotstudio.com","Delete","Global")
add item to list(%urls,"http://www.bing.com","Delete","Global")
add item to list(%urls,"http://www.ebay.com","Delete","Global")
add item to list(%urls,"http://www.stackoverflow.com","Delete","Global")
divider
plugin command("Smartthreads.dll", "Thread control normal", "Test", $list total(%urls), 3, "No")
define Test(#pos) {
    set(#result,$plugin function("HTTP post.dll", "$http get", $list item(%urls,#pos), $plugin function("HTTP post.dll", "$http useragent string", "Random"), "", "", 10),"Local")
    comment("SCRAPE THE PAGE TITLE")
    alert($find regular expression(#result,"\\<title\\b[^>]*\\>\\s*(?<Title>[\\s\\S]*?)\\</title\\>"))
}

here you go thanks