Jump to content
UBot Underground

[FREE PLUGIN] Threads Counter - UBot v4 Threading Fixed


Recommended Posts

Im simply just doing ' next list item ' of an already scraped list and it seems to completely skip it. even if i use 'thread safe container' on that next list item command, the thread loop count and the list position are completely off meaning its still getting skipped. I guess ubot simply cant handle more than 50 threads and thread safe container doesnt even work. thread count works most of the time, but i still sometimes get -negative count 

Link to post
Share on other sites
  • Replies 160
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

As you may be aware threading in UBot v4 is not safe and may not work as expect under certain conditions, which would result in threads slowly dying out, so at the end, you only get left with 1 active

Awesome plugin UBotDev

You are not using the plugin from the title, instead you are using the plugin that Blumi 40 keeps posting here (Blumi40, can you please open another thread for that plugin instead of discussing it her

There is no problem with the threads counter plugin. But I am sure if you are using threads along with add to list there are problems with your scraping. The issue is even though the thread counter helps there is still a problem when adding data to a list at the exact same time as another thread. Run my code and you will see that not all of the items are being added to the list when running threads.

Just tried out your code and i see what you mean... this is a HUGe problem and the fact that 'thread safe container' doesnt work at all or even makes it worse means there is no solution to this..... LAME i hate ubot

Link to post
Share on other sites

There is no problem with the threads counter plugin. But I am sure if you are using threads along with add to list there are problems with your scraping. The issue is even though the thread counter helps there is still a problem when adding data to a list at the exact same time as another thread. Run my code and you will see that not all of the items are being added to the list when running threads.

That's very interesting. Can confirm that it's happening for me as well.

And in V5 the complete UI or compiled bot is crashing as well if I run 500 threads for example.

 

I opened a tracker for that issue:

http://tracker.ubotstudio.com/issues/368

 

Dan

Link to post
Share on other sites

Here's an idea how to "fix" it:

 

 

ui drop down("Loops:", "100,200,300,400,500,600,700,800,900,1000", #Loops)
comment("After running the script the total count should be exactly the same number as the loops set in in the ui.")
ui stat monitor("Open Threads:", #thread count)
ui stat monitor("Total:", $table total rows(&numbers))
clear table(&numbers)
set(#thread count, $plugin function("Threads Counter.dll", "threads counter", "reset"), "Global")
set(#loopcounter, 0, "Global")
loop(#Loops) {
    set(#thread count, $plugin function("Threads Counter.dll", "threads counter", "increment"), "Global")
    test threads($random text(50), #loopcounter)
    increment(#loopcounter)
}
loop while($comparison(#thread count, ">", 0)) {
    wait(0.5)
}
stop script
define test threads(#number, #rownumber) {
    thread {
        wait($rand(1, 3))
        comment("The problem is that add item to list cant add data at the same exact time as another thread. Or else it doesnt add an item to the list. This is a huge problem.")
        set table cell(&numbers, #rownumber, 0, #number)
        set(#thread count, $plugin function("Threads Counter.dll", "threads counter", "decrement"), "Global")
    }
}

 

 

Cheers

Dan

  • Like 1
Link to post
Share on other sites

So using tables with threads works fine but lists dont? 

 

It should be fixed and its just not add item to list its also next list item and i assume any list function. The problem is.. eddie quit... and ubot 4 is no longer updated... so seems we are screwed. 

Link to post
Share on other sites

There is no problem with the threads counter plugin. But I am sure if you are using threads along with add to list there are problems with your scraping. The issue is even though the thread counter helps there is still a problem when adding data to a list at the exact same time as another thread. Run my code and you will see that not all of the items are being added to the list when running threads.

Nice catch Gogetta. I've never noticed that, I guess it's because I always use tables in such cases, but yeah, it looks like global lists are also not thread safe. :/

Link to post
Share on other sites

@Dan I would set the random wait($rand(1, 3)) to a fixed time.

As I believe it only adds confusion and puts off the inevitable if it’s going to happen.

 

Also the random text I think is a mistake as more the one row could receive the same data and you will never know, So you could possibly just trade problems.

 

Having said this I find it hard to believe ubot will operate correctly at one hundred or more threads.  As if you look back to version 3 the thread command was added so we could close popup windows. :)  It was 3.5 before we got local variables and threads run as separate instances.  It was at this point in time version 4 was released as no one could multi-thread with V3.?.? as data was being lost all over the place. Then it was a six month wait before V4 was stable.

Link to post
Share on other sites

Would that be an option? But add to list should be fixed nevertheless.

 

 

Dan

 

Sure, this is more than just an option for now. I see this as being the only way to ensure my scrapers are returning the correct amount of data. This bug where list and variables are canceling each other out needs to be fixed asap. But thanks for the workaround. I didnt even think to try using a table since all of the other commands I tried had the same problem.

 

Add to list, remove from list, append to file, all of these commands become useless once you start using hundreds of threads. I am not sure why this hasn't been fixed yet and hopefully Seth will send out an update for both 4 and 5 since most of us are still using V.4.

 

 

Nice catch Gogetta. I've never noticed that, I guess it's because I always use tables in such cases, but yeah, it looks like global lists are also not thread safe. :/

 

Right, I will be using tables from here on out, well at least until the add to list has been fixed. I was gonna ask you if there is a way to use the threads counter locally, or if you have any plans to make it work locally?

Link to post
Share on other sites

I thought they no longer have a developer? That's why there hasnt been any updates in months. 

 

The thing with tables is, if im using 'increment' to go to the next row of a table, isnt increment also not thread safe? 

Link to post
Share on other sites

I thought they no longer have a developer? That's why there hasnt been any updates in months. 

 

The thing with tables is, if im using 'increment' to go to the next row of a table, isnt increment also not thread safe? 

You should increment row number outside the thread command and pass the number as parameter into thread/define command that wraps it. The idea  is show with 1st example on 1st post (THREAD START define command): http://www.ubotstudio.com/forum/index.php?/topic/15441-free-plugin-threads-counter-ubot-v4-threading-fixed/

Link to post
Share on other sites

SO if i pass a parameter outside the thread command into a thread command, it stays local to that thread?

It's actually local to define command where you also pass it, and not thread, that's why you need a wrapper define command around the thread with it.

Link to post
Share on other sites

hmmm i seem to be having issues, but its late at night. WIll try tomorrow and see how it works. SO basically passing a variable to a define keeps it local and in that way instead of next list item, i can use a variable with an increment and list item to successfully go to the next list item, correct?

Link to post
Share on other sites

 

UbotDev

 

Till today I only had one bot which was crashing constantly when running on 100 threads, and the solution was to close the bot after a while and re-open and restart it from where where it left off. This way I was able to process millions of inputs without any problem...

 

UbotDev..? Does this work for  JavaScript  heavy  webpages too or are they mainly for '$read file' kind of scrapes?

 

Say  , in order to scrape all comments off a Youtube video page - may require  'clicking' the  'show more'  button (sometimes multiple times) - which executes javascript on the page.

 

Are such actions  possible/stable at 100 threads?

I had tried the above - just with a few threads and what would happen is that if the number of comments  happened to be   exceptionally high - then Ubot would  go into an endless loop  after sometime.. I mean Ubot would not  respect  the 'exit loop' logic.

 

so I  was wondering if you are able to handle this kind of action at high thread numbers too?

 

Thanks.

Link to post
Share on other sites

UbotDev..? Does this work for  JavaScript  heavy  webpages too or are they mainly for '$read file' kind of scrapes?

 

Say  , in order to scrape all comments off a Youtube video page - may require  'clicking' the  'show more'  button (sometimes multiple times) - which executes javascript on the page.

 

Are such actions  possible/stable at 100 threads?

I had tried the above - just with a few threads and what would happen is that if the number of comments  happened to be   exceptionally high - then Ubot would  freeze half way through the process!

 

so I  was wondering if you are able to handle this kind of action at high thread numbers too?

 

Thanks.

It worked via UBot browser but the page was not JS heavy.

 

Such action on so called "infinite pages" are possible if you can keep browser sized bellow 1-1.5 GB, else browser will crash (as mentioned here). I think the only way around would be to remove the items you already scraped, and keep browser memory size low like that.

 

Again, the problem is not related to threads, threads only make things worse because chances for browser crashing are greater.

  • Like 1
Link to post
Share on other sites

 

I think the only way around would be to remove the items you already scraped

 

Wow! How did I miss this little gem on that thread?  Thanks UbotDev!

 

Link to post
Share on other sites
  • 3 weeks later...

I am using the same exact script from the post #1 (proxies test). but I have to use "in shared browser" instead of "in new browser" since I have to stay logged in (Im scraping info from facebook).  The problem actually is that my bot opens the same page all the time, and does not scrapes what I want.  This issue only happens when I use "in shared browser", does it make sense?

 

 

Luis Carlos

Link to post
Share on other sites

I am using the same exact script from the post #1 (proxies test). but I have to use "in shared browser" instead of "in new browser" since I have to stay logged in (Im scraping info from facebook).  The problem actually is that my bot opens the same page all the time, and does not scrapes what I want.  This issue only happens when I use "in shared browser", does it make sense?

 

 

Luis Carlos

Are you sure that's all you changed? Please share your code, else it's almost impossible to help you.

Link to post
Share on other sites

I have a quick question.

 

 

Is it ok to use the plugin directly within a stat monitor?

ui stat monitor("Open Threads:", $plugin function("Threads Counter.dll", "threads counter", "read"))

 

And what's the reason for using a Function instead of a command here?

set(#threadcount, $plugin function("Threads Counter.dll", "threads counter", "increment"), "Global")

Wouldn't it be "better" to change that to a command, so that we can avoid using variables at all?

Like:

plugin command("Threads Counter.dll", "threads counter", "Increment")

 

If I need it in a variable I could always do:

set(#threadcount, $plugin function("Threads Counter.dll", "threads counter", "read"), "Global")

 

 

 

Thanks in advance for your help

Dan

Link to post
Share on other sites

I have a quick question.

 

 

Is it ok to use the plugin directly within a stat monitor?

ui stat monitor("Open Threads:", $plugin function("Threads Counter.dll", "threads counter", "read"))

 

And what's the reason for using a Function instead of a command here?

set(#threadcount, $plugin function("Threads Counter.dll", "threads counter", "increment"), "Global")

Wouldn't it be "better" to change that to a command, so that we can avoid using variables at all?

Like:

plugin command("Threads Counter.dll", "threads counter", "Increment")

 

If I need it in a variable I could always do:

set(#threadcount, $plugin function("Threads Counter.dll", "threads counter", "read"), "Global")

 

 

 

Thanks in advance for your help

Dan

I don't think it's. I never use UI like that, I always use variables there.

 

It is a function so that you don't have to call "read" command after every increment, for example for displaying counter in UI.

Link to post
Share on other sites

Just tested:

ui stat monitor("Open Threads:", $plugin function("Threads Counter.dll", "threads counter", "read"))

 

And it works fine :-)

I know it works...but I don't think that's efficient, since UI always needs to execute function that way vs. only reading variable (and I think that happens pretty often since UI is constantly updated)...

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...