Jump to content
UBot Underground

Recommended Posts

I am building a bot that navigates to 3 different sites, logs in (with 3 different logins & passwords), scrapes data, and posts the information. The bot works well, but I would like to speed things up by threading and doing all 3 processes at the same time.

 

I am using defines for each site as follows:

 

Define site 1

     navigate to site

     enter login data

     scrape page

     save scraped data to list

Define site 2

     navigate to site

     enter login data

     scrape page

     save scraped data to list

Define site 3

     navigate to site

     enter login data

     scrape page

     save scraped data to list

 

For each define, I tried:

 

Define site 1

Thread

     in new browser

          navigate to site

          enter login data

          scrape page

          save scraped data to list

 

And although this technique performs each define in a new browser, the process takes place 1 define at a time and is not any faster. How do I use threading to perform these 3 defines simultaneously?

 

Link to post
Share on other sites

Hi there are may different ways to do multithreading, this is a simple way to code it, I did this quick but is probably what you need.

 

Start()
define Start {
    thread {
        in new browser {
            Define One()
        }
    }
    thread {
        in new browser {
            Define Two()
        }
    }
    thread {
        in new browser {
            Define Three()
        }
    }
}
define Define One {
    navigate("http://google.com", "Wait")
    wait(5)
}
define Define Two {
    navigate("http://google.com", "Wait")
    wait(5)
}
define Define Three {
    navigate("http://google.com", "Wait")
    wait(5)
}
  • Like 2
Link to post
Share on other sites

Yes that should be correct. Here is a threading example that works maybe it will help you:

 

startnow()
define startnow {
    google()
    yahoo()
}
define google {
    thread {
        in new browser {
            navigate("http://google.com", "Wait")
            wait(2)
            type text(<name="q">, "modern marvels", "Standard")
            wait for browser event("Everything Loaded", "")
            wait(10)
        }
    }
}
define yahoo {
    thread {
        in new browser {
            navigate("http://www.yahoo.com/", "Wait")
            wait(2)
            type text(<class="input-query input-long med-large  ">, "top gear", "Standard")
            wait(.5)
            click(<id="search-submit">, "Left Click", "No")
            wait for browser event("Everything Loaded", "")
            wait(10)
        }
    }
}
  • Like 2
Link to post
Share on other sites

it is good post and I've encountered same problem with thread command, in my opinion current implementation of thread command is only useful for ddos some site or things like this, because if I want to perform some task in let's say 20 threads I would need to create 20 define's for each tread - it is insane!

And what if I want 100 threads :o ?  

Link to post
Share on other sites
it is good post and I've encountered same problem with thread command, in my opinion current implementation of thread command is only useful for ddos some site or things like this, because if I want to perform some task in let's say 20 threads I would need to create 20 define's for each tread - it is insane!

And what if I want 100 threads :o ?  

 

I'm not really sure that is the issue that you have, but if you're stating that you need to re-DEFINE your custom commands/functions for each thread, then I'm afraid YOU are doing something wrong there. 

(no offense meant, just trying to help you)

 

My guess is you are using GLOBAL variables in your code, all over, including the DEFINEs .. which, if it were true, might indeed mix up values and render you bots useless, in my opinion.

 

On the other hand, if your variables are set as LOCAL in each DEFINE, you can repeatedly call the same function/command in each thread and it should keep the values confined to that thread, thus, w/o mixing them with other threads running concurrently, there is no room left for errors anymore

(at least from THIS issue << which I observed to be a common mistake in people's coding)

 

Hope this helps...

Link to post
Share on other sites

You right ! I'm using Global all other the place, will try to switch to local for each define. Thanks for the help!

I'm not really sure that is the issue that you have, but if you're stating that you need to re-DEFINE your custom commands/functions for each thread, then I'm afraid YOU are doing something wrong there. 
(no offense meant, just trying to help you)

 

My guess is you are using GLOBAL variables in your code, all over, including the DEFINEs .. which, if it were true, might indeed mix up values and render you bots useless, in my opinion.

 

On the other hand, if your variables are set as LOCAL in each DEFINE, you can repeatedly call the same function/command in each thread and it should keep the values confined to that thread, thus, w/o mixing them with other threads running concurrently, there is no room left for errors anymore
(at least from THIS issue << which I observed to be a common mistake in people's coding)

 

Hope this helps...

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...