How Do I Handle Multi Threading With A Variable?

[Fer 26]

Lets say I have a multi threaded bot, and they all have to scrape something from pages.

And I start off with a variable, lets say #PageNumber and its value is 1.

 

Now, each thread needs to increment the global variable by 1, and then visit the page for example: http://somewebsite.com/index.php?pagenumber=#PageNumber
But how do I know if this is accurate? I am pretty sure threads would mess up and for example it would increment 3 times before one thread actually navigates to the page, causing my bot to skip a page/pages.

 

Any ideas? Also if I wasn't clear enough let me know.

 

 

(Yes, Next list item is more accurate but I can't use that here now can I . Well, maybe but doesn't sound efficient saving numbers in a list. But would work since $next list item would be called in the same command to navigate, so you know it can't screw up.)

[Brutal 164]

One option would be to use the "list item" function with your #pagenumber_var incrementing each time the thread command runs.

[cүвεя_נυηкιε 68]

+1 for Brutals option

 

as for :

I am pretty sure threads would mess up and for example it would increment 3 times before one thread actually navigates to the page

 

 

place a "Wait For Browser Event" after the navigate, maybe even a 2 second wait after that, you know to make sure the page has loaded before it moves on to the next,

Or a wait for a specific element that you know will be on the page before it continues, 

 

or both even lol, .. 

 

Im a fan of adding stuff to lists, 

purely so you can either call with $next list item, or set the next list item to a variable and do it that way, 

all ways are pretty effective

 

 

 

 

CJ

[Fer 26]

+1 for Brutals option

 

as for :

 

place a "Wait For Browser Event" after the navigate, maybe even a 2 second wait after that, you know to make sure the page has loaded before it moves on to the next,

Or a wait for a specific element that you know will be on the page before it continues, 

 

or both even lol, .. 

 

Im a fan of adding stuff to lists, 

purely so you can either call with $next list item, or set the next list item to a variable and do it that way, 

all ways are pretty effective

 

 

 

 

CJ

 

One option would be to use the "list item" function with your #pagenumber_var incrementing each time the thread command runs.

 

Thanks guys, but unfortunately that wouldn't solve my problem as I still have to increment the variable.

Let me put it this way

 

Thread 1 > Increment > Navigate

Thread 2 > Increment > Navigate

 

Lets say, that Thread 2 is faster than Thread 1. 

 

And that it goes like this:

 

1. Thread 2 would increment the variable

2. Thread 1 would increment the variable

3. Thread 2 would navigate

4. Thread 1 would navigate

 

Then I would have 2 threads navigating at the same place, because one didn't manage to Increment + Navigate instantly. 

So I'm talking about a delay between increment and navigate.

 

If we look at next list item, then its accurate because its being called in the same command when navigating.

But if I had a list item referring to a variable which I still increment it would be futile since its the increment part that goes wrong.

 

 

An option would be to add 1-5000 in a list, but sounds pretty inefficient and we know that UBot can't handle lists larger than a certain amount.

[cүвεя_נυηкιε 68]

Thanks guys, but unfortunately that wouldn't solve my problem as I still have to increment the variable.

Let me put it this way

 

Thread 1 > Increment > Navigate

Thread 2 > Increment > Navigate

 

Lets say, that Thread 2 is faster than Thread 1. 

 

And that it goes like this:

 

1. Thread 2 would increment the variable

2. Thread 1 would increment the variable

3. Thread 2 would navigate

4. Thread 1 would navigate

 

Then I would have 2 threads navigating at the same place, because one didn't manage to Increment + Navigate instantly. 

So I'm talking about a delay between increment and navigate.

 

If we look at next list item, then its accurate because its being called in the same command when navigating.

But if I had a list item referring to a variable which I still increment it would be futile since its the increment part that goes wrong.

 

 

An option would be to add 1-5000 in a list, but sounds pretty inefficient and we know that UBot can't handle lists larger than a certain amount.

 

 

Hi Fer

 

 

To be honest $next list item should be fine, 

I have a bot that visits 72 sites over and over infinitely, and even running 18 threads ive never seem it miss a page or load a duplicate,

 

 

I just read through your OP, maybe you could explain a little further in case we are just not understanding properly (i get this a lot hahaha)

for example, ..

it would increment 3 times before one thread actually navigates to the page

 

why will it ?

what i mean is why is that important, if you have a thread limit and each thread is called then each will do what its supposed to so why does it matter ?

 

 

also 

causing my bot to skip a page/pages.

 

If the thread was called and running then it wont skip pages, it may load pages in an odd order if some pages load quicker or slower than others, 

but is it "critical" that the pages load in a "specific" order ?

 

 

 

let us know and im sure it will help find a suitable solution

 

 

This reminds me of a project i had a few weeks ago, twisted my head for weeks over a solution, 

in the end i created my own Thread ID system via table so that each thread had its own data to read from, ... 

 

But I sort of get what you mean as using a Global variable for thread purposes will always be a pain in the A** lol

 

 

 

 

Cheers

CJ