kev123 132 Posted February 23, 2015 Author Report Share Posted February 23, 2015 pm sent Quote Link to post Share on other sites
UBOTEM 18 Posted March 20, 2015 Report Share Posted March 20, 2015 Just bought Have to sleep, but will try out tomorrow Hopefully it may solve an issue I currently have with HTTP Get activated is stopping my threads to fire to max value. I was using the free thread counter and script from ubotdev but seems to have issue when switch to http get instead of browse and use $documentText. Anyway will try out and report if fixed issue, as may solve for others two Looks simple easy to setup GREAT JOB Quote Link to post Share on other sites
UBOTEM 18 Posted March 20, 2015 Report Share Posted March 20, 2015 [FREE] Scrape Framework Here is a free scraping framework for you guys. It's usingthe smartthread plugin and the large data plugin from Kev. DOWNLOAD Hello Dan, Link does not work anymore?Can you send me a PM or something please Quote Link to post Share on other sites
UBOTEM 18 Posted March 20, 2015 Report Share Posted March 20, 2015 (edited) Umm... maybe I did something wrong? but has slowed down my bot! Stripped my bot of old thread code. moved thread process into a define without thread. then placed plugin command("Smartthreads.dll", "Thread control normal", "THREAD START", #COUNT, #THREADS Max, "No") where my old multi thread loop was. tested and working fine, and shows all threads are being used.but seemed slower! (when was hoping for faster) not changed anything else just swapped over to your smart threads. So I opened both versions of my compiled bot, new one with smart threads, and old previous. Set both up to do same job, with only 2 threads, started yours 2 min before my old multi-threaded version, and about 5 min in my old one took over! The main issue with my previous version of my bot, that was hoping would fix issue (which sort of has) I have a switch to change scrape method like: if($comparison(#input_method, "=", "HTTP Get")) { then { set(#http_get, $plugin function("HTTP post.dll", "$http get", #var_url, #var_userAgent, #var_referer, "", "{#var_httpGetTimeout}000"), "Local") } else { in new browser { set user agent(#var_userAgent) set referrer(#var_referer) navigate(#var_url, "Wait") wait for browser event("Everything Loaded", #var_httpGetTimeout) set(#http_get, $document text, "Local") wait(1) close page } } } And when I switch to HTTP it never fired threads above 13, yet browser mode always max-ed to whatever set. Your plugin has defiantly fixed that as can see that the threads are being used, but seems to be slower? my define for thread is quite simple.Grabs html using either HTTP or BROWSER as above.runs quite a few xpath and regex on the html, about 20 checks at moment.saves to a normal ubot table. (know large tables will help and is next on my list to do, after this issue)Love your plugin though, very simple, maybe i'm doing something wrong? My plan is to have HTTP threads upto 200 (currently seems very slow at 10 with your plugin, and faster without but never goes above 13ish) Am quite competent, but maybe missing something out? Not tried the proxies yet, but that looks interesting also Edited March 20, 2015 by UBOTEM Quote Link to post Share on other sites
UBOTEM 18 Posted March 20, 2015 Report Share Posted March 20, 2015 BTW switching code over was quite straight forward, and a lot less code Just need to find out why it has slowed down? Quote Link to post Share on other sites
Bot-Factory 602 Posted March 20, 2015 Report Share Posted March 20, 2015 BTW switching code over was quite straight forward, and a lot less code Just need to find out why it has slowed down? Well the question is.. did it really slow down?Not running the same amount of threads in parallel doesn't automatically mean it is slower now. The http request takes a lot less time compared to a browser command. So the threads will finish in a lot less time. And that could be the reason why the thread trashhold can not be reached. Just add a wait(1)to the end of the thread define. And you will see that the thread total will increase. But increasing the threads isn't the goal right.Completing everything as fast as possible. And if your threads finish very quickly, it's a matter of your PC power on how many threads it can start. Dan Quote Link to post Share on other sites
UBOTEM 18 Posted March 20, 2015 Report Share Posted March 20, 2015 Well the question is.. did it really slow down?Not running the same amount of threads in parallel doesn't automatically mean it is slower now. The http request takes a lot less time compared to a browser command. So the threads will finish in a lot less time. And that could be the reason why the thread trashhold can not be reached. Just add a wait(1)to the end of the thread define. And you will see that the thread total will increase. But increasing the threads isn't the goal right.Completing everything as fast as possible. And if your threads finish very quickly, it's a matter of your PC power on how many threads it can start. Dan Hi Dan, Thanks for help, maybe I am not understanding or wires crossed :/ But before when I used the old threading method and http threads never maxed out (so that makes sense that they finished quicker, but still expect the threads to fire out to 20+ or even 200+ and crash!)I'm sure I tried to add a wait at that point and still never maxed out. (will try a 10 sec wait and see if makes any difference) Anyway moving on.... I switched to smartthreads and now it does fire to the max, but seems to freeze my lappy (running quad core 8gb ram). My test was.... old version vs smartthread version on only 2 http threads each, and old version seemed/was to be faster at process of 500 items. Its really confused me, as in old threaded version I would still expect it to max out! Confused me as does not seem logical As for parallel threads, surely both multithread is parallel? Also Dan, can you show us the free framework you shared with smartthreads & big lists? I need to be able to use http with smarthreads and big lists my target is upto 200 threads, does this seem reasonable? Quote Link to post Share on other sites
MMD 7 Posted March 21, 2015 Report Share Posted March 21, 2015 Can i run two "Thread control normal" command at the same time without problems? Havent tested it but i hope its not problem. Quote Link to post Share on other sites
kev123 132 Posted March 21, 2015 Author Report Share Posted March 21, 2015 ubotem Have you got a code sample? MMD no because you would need to use ubot internal threads to accomplish this and that could cause problems.You can however use advanced add commands. 1 Quote Link to post Share on other sites
Bot-Factory 602 Posted March 21, 2015 Report Share Posted March 21, 2015 Hi Dan, Thanks for help, maybe I am not understanding or wires crossed :/ But before when I used the old threading method and http threads never maxed out (so that makes sense that they finished quicker, but still expect the threads to fire out to 20+ or even 200+ and crash!)I'm sure I tried to add a wait at that point and still never maxed out. (will try a 10 sec wait and see if makes any difference) Anyway moving on.... I switched to smartthreads and now it does fire to the max, but seems to freeze my lappy (running quad core 8gb ram). My test was.... old version vs smartthread version on only 2 http threads each, and old version seemed/was to be faster at process of 500 items. Its really confused me, as in old threaded version I would still expect it to max out! Confused me as does not seem logical As for parallel threads, surely both multithread is parallel? Also Dan, can you show us the free framework you shared with smartthreads & big lists? I need to be able to use http with smarthreads and big lists my target is upto 200 threads, does this seem reasonable? If you want to hear my honets oppinion... Threading with a high number of threads will give you a lot of strange errors with Ubot studio.But not because of the smartthread plugin or anything else. The main reason is the Ubot Studio API. There are a couple of known "bugs" Just look into the tracker and view the API bug reports. I would not waste a lot of time with optimization. There is no way for us to optimize that behavior from the outside. I used an ubot scraper to scrape 5 million urls. But instead of optimizing everything. I just ran 6 bots on 3 VPS servers. And at the end, I just combined the output of all 6 bots and removed duplicates. Job done. Scale horizontal (more bots on more servers)And not vertical (more threads on single bot and single server) Just my 2 centsDan Quote Link to post Share on other sites
UBOTEM 18 Posted March 25, 2015 Report Share Posted March 25, 2015 OK quick question.... Say I detect that a proxy is bad within my thread/define, how do I switch proxy? Or do we have to mark thread bad, to be redone? Quote Link to post Share on other sites
UBOTEM 18 Posted March 25, 2015 Report Share Posted March 25, 2015 @dan thanks for advice Also would like to know more about VPS and how to run on there! (also would this mean my Mac customers could use?) Quote Link to post Share on other sites
Bot-Factory 602 Posted March 26, 2015 Report Share Posted March 26, 2015 @dan thanks for advice Also would like to know more about VPS and how to run on there! (also would this mean my Mac customers could use?)I usevultr.com You can get a windows based server from them to run all your bots. And yes, you can connect to that server from a mac (via RDP) as well.But that's probably not something for your customers. Because you would have to provide them access to your server. I mean, it's possible. But that's a configuration and scalability issue. Depending on what your bot is doing and how many users you are looking for. if this is just something you sell for a high price to 10-30 users, a terminal server based solution could be a possibility. But definately nothing for a larger scale scenario. In such a case I would look into developing a web based frontend for your customers. And let that frontend connect to your bot on the backend server. The communication plugin offers some features to get you started.If that is the kind of scenario you are looking for. CheersDan Quote Link to post Share on other sites
UBOTEM 18 Posted March 26, 2015 Report Share Posted March 26, 2015 (edited) @dan thanks for info, not quite ready yet for it but will be aiming for a website front-end, and a server hosting bots. User requests job, job gets queued, email user with link to data Edited March 26, 2015 by UBOTEM Quote Link to post Share on other sites
Hightower 2 Posted April 15, 2015 Report Share Posted April 15, 2015 smartthreads plugin is not able to perform required task. I gave it list of 40 list items but it does not perform action on each. it is missing some the list items... help please Quote Link to post Share on other sites
Bot-Factory 602 Posted April 15, 2015 Report Share Posted April 15, 2015 smartthreads plugin is not able to perform required task. I gave it list of 40 list items but it does not perform action on each. it is missing some the list items... help please Please provide some example code and more details. It's quite hard to understand where you run into problems that way.I have smart thread plugin running in a couple of scenarios without problems. But you always have to be careful with ubot and threading. There are some limitations.If threads execute to quickly, or are started and finished within milliseconds, you will always see strange behavior. But that is not related to smartthread plugin. That's just Ubot...You would see the same behavior with normal threading. Also if you use other plugin commands within a thread, it could cause issues. I also see it very often that people apply "old" (V4) concepts to ubot studio and the latest updates. Like using http container with http plugin. Or using local variable plugin for local variables. And a lot of that stuff is from the v4 days and don't work in a similar way with v5. So there is really a lot of stuff that can go wrong here. And without knowing more details, it's not possible to help you. Dan Quote Link to post Share on other sites
kev123 132 Posted April 16, 2015 Author Report Share Posted April 16, 2015 as dan said code example if you can thanks Quote Link to post Share on other sites
UBOTEM 18 Posted May 1, 2015 Report Share Posted May 1, 2015 Ahh help, having issues where only 1 set of threads firing out?! Loops @ 6Threads @ 4 plugin command("Smartthreads.dll", "Thread control normal", "main_prog", #thread_loops, #input_maxThreads, "No") 4 threads fire up!Also placed an alert at the end of the command 'main_prog' so can see they hit that, but next two do not run, and my bot hangs Using 1.3 I think Any help v much appreciated Quote Link to post Share on other sites
UBOTEM 18 Posted May 1, 2015 Report Share Posted May 1, 2015 Ahh help, having issues where only 1 set of threads firing out?! Loops @ 6Threads @ 4 plugin command("Smartthreads.dll", "Thread control normal", "main_prog", #thread_loops, #input_maxThreads, "No") 4 threads fire up!Also placed an alert at the end of the command 'main_prog' so can see they hit that, but next two do not run, and my bot hangs Using 1.3 I think Any help v much appreciated Solved myself Thought was going mad.... So checked threads in use, and realised that I used a normal thread on startup.Removed this and smartthreads is now doing its job! Quote Link to post Share on other sites
BeerNut 25 Posted May 19, 2015 Report Share Posted May 19, 2015 I have just incorporated this plugin into one of my more simple bots. But I am having a strange error where pressing the uBot stop button does not stop the bot anymore. The program will continue to go though the list and keeping going until it finishes the task. Its basically loading a list, then looping the list total and then doing a thing. I am not passing the list to the smart thread system just yet. So inside the thread its pulling next list item. But I am passing the loop (list total) directly to the smart thread. So it knows it needs to loop 1600 times and will continue to loop 1600 times even if I press the uBot stop button. It looks like I need to make a new UI to use this plugin and only use your stop and start buttons? Or is this just a bug? Edit: I should mention that I am using Thread control Advanced add, Loop Number, list total (%list) as the loop number, and no proxies. Using uBot 4. Edit 2: I watched the videos and make a new thread using the Thread control Advanced add, List, %list, no proxies. Once again if I press uBot stop the program continues though the list until it finishes. I also tested in uBot 5 and it has the same issue. Quote Link to post Share on other sites
kev123 132 Posted May 19, 2015 Author Report Share Posted May 19, 2015 hi you are correct you need to use stop and start button, there is a command to stop the plugin. thanksKev123 Quote Link to post Share on other sites
SaHiN 14 Posted June 12, 2015 Report Share Posted June 12, 2015 Hi kev, working aymen http post command multi thread? Quote Link to post Share on other sites
kev123 132 Posted June 12, 2015 Author Report Share Posted June 12, 2015 hey sorry for delay in response yes it will work fine if you have problems post here and i'll take a look, Quote Link to post Share on other sites
SaHiN 14 Posted June 12, 2015 Report Share Posted June 12, 2015 hey sorry for delay in response yes it will work fine if you have problems post here and i'll take a look, please sample code? Quote Link to post Share on other sites
kev123 132 Posted June 14, 2015 Author Report Share Posted June 14, 2015 comment("add some sites to list just for example") clear list(%urls) add item to list(%urls,"http://www.ubotstudio.com","Delete","Global") add item to list(%urls,"http://www.bing.com","Delete","Global") add item to list(%urls,"http://www.ebay.com","Delete","Global") add item to list(%urls,"http://www.stackoverflow.com","Delete","Global") divider plugin command("Smartthreads.dll", "Thread control normal", "Test", $list total(%urls), 3, "No") define Test(#pos) { set(#result,$plugin function("HTTP post.dll", "$http get", $list item(%urls,#pos), $plugin function("HTTP post.dll", "$http useragent string", "Random"), "", "", 10),"Local") comment("SCRAPE THE PAGE TITLE") alert($find regular expression(#result,"\\<title\\b[^>]*\\>\\s*(?<Title>[\\s\\S]*?)\\</title\\>")) } here you go thanks Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.