jomark3 25 Posted January 25, 2013 Report Share Posted January 25, 2013 I am building a bot that navigates to 3 different sites, logs in (with 3 different logins & passwords), scrapes data, and posts the information. The bot works well, but I would like to speed things up by threading and doing all 3 processes at the same time. I am using defines for each site as follows: Define site 1 navigate to site enter login data scrape page save scraped data to listDefine site 2 navigate to site enter login data scrape page save scraped data to listDefine site 3 navigate to site enter login data scrape page save scraped data to list For each define, I tried: Define site 1Thread in new browser navigate to site enter login data scrape page save scraped data to list And although this technique performs each define in a new browser, the process takes place 1 define at a time and is not any faster. How do I use threading to perform these 3 defines simultaneously? Quote Link to post Share on other sites
LazyBotter 188 Posted January 25, 2013 Report Share Posted January 25, 2013 Hi there are may different ways to do multithreading, this is a simple way to code it, I did this quick but is probably what you need. Start() define Start { thread { in new browser { Define One() } } thread { in new browser { Define Two() } } thread { in new browser { Define Three() } } } define Define One { navigate("http://google.com", "Wait") wait(5) } define Define Two { navigate("http://google.com", "Wait") wait(5) } define Define Three { navigate("http://google.com", "Wait") wait(5) } 2 Quote Link to post Share on other sites
HelloInsomnia 1103 Posted January 25, 2013 Report Share Posted January 25, 2013 Yes that should be correct. Here is a threading example that works maybe it will help you: startnow() define startnow { google() yahoo() } define google { thread { in new browser { navigate("http://google.com", "Wait") wait(2) type text(<name="q">, "modern marvels", "Standard") wait for browser event("Everything Loaded", "") wait(10) } } } define yahoo { thread { in new browser { navigate("http://www.yahoo.com/", "Wait") wait(2) type text(<class="input-query input-long med-large ">, "top gear", "Standard") wait(.5) click(<id="search-submit">, "Left Click", "No") wait for browser event("Everything Loaded", "") wait(10) } } } 2 Quote Link to post Share on other sites
jomark3 25 Posted January 25, 2013 Author Report Share Posted January 25, 2013 Thanks guys, worked like a charm!!! Quote Link to post Share on other sites
bestmacros 60 Posted January 28, 2013 Report Share Posted January 28, 2013 it is good post and I've encountered same problem with thread command, in my opinion current implementation of thread command is only useful for ddos some site or things like this, because if I want to perform some task in let's say 20 threads I would need to create 20 define's for each tread - it is insane!And what if I want 100 threads ? Quote Link to post Share on other sites
VaultBoss 310 Posted January 28, 2013 Report Share Posted January 28, 2013 it is good post and I've encountered same problem with thread command, in my opinion current implementation of thread command is only useful for ddos some site or things like this, because if I want to perform some task in let's say 20 threads I would need to create 20 define's for each tread - it is insane!And what if I want 100 threads ? I'm not really sure that is the issue that you have, but if you're stating that you need to re-DEFINE your custom commands/functions for each thread, then I'm afraid YOU are doing something wrong there. (no offense meant, just trying to help you) My guess is you are using GLOBAL variables in your code, all over, including the DEFINEs .. which, if it were true, might indeed mix up values and render you bots useless, in my opinion. On the other hand, if your variables are set as LOCAL in each DEFINE, you can repeatedly call the same function/command in each thread and it should keep the values confined to that thread, thus, w/o mixing them with other threads running concurrently, there is no room left for errors anymore(at least from THIS issue << which I observed to be a common mistake in people's coding) Hope this helps... Quote Link to post Share on other sites
bestmacros 60 Posted January 28, 2013 Report Share Posted January 28, 2013 You right ! I'm using Global all other the place, will try to switch to local for each define. Thanks for the help!I'm not really sure that is the issue that you have, but if you're stating that you need to re-DEFINE your custom commands/functions for each thread, then I'm afraid YOU are doing something wrong there. (no offense meant, just trying to help you) My guess is you are using GLOBAL variables in your code, all over, including the DEFINEs .. which, if it were true, might indeed mix up values and render you bots useless, in my opinion. On the other hand, if your variables are set as LOCAL in each DEFINE, you can repeatedly call the same function/command in each thread and it should keep the values confined to that thread, thus, w/o mixing them with other threads running concurrently, there is no room left for errors anymore(at least from THIS issue << which I observed to be a common mistake in people's coding) Hope this helps... Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.