Jump to content
UBot Underground

What Did I Do Wrong? Multi-Threaded Bot Issue


Recommended Posts

The bot loops through a list of URLs and searches each site for a keyword. Then it saves the results in the CSV.

 

Any idea what I'm doing incorrectly?

 

The bot skips some of the first list items, and each thread loads the same URL.. I tried fixing it for hours, but no luck.

 

TAB 1 CODE:

on load("Bot Loaded") {
    navigate("about:blank", "Wait")
    alert("Please Choose a CSV File to Process")
}
initialSetup()
ui open file("CSV List To Process:", #list)
ui button("Click to Save Updated CSV File") {
    save to file(#list, &url_table)
}
ui stat monitor("Total Pages to Scan:", #ListMax)
ui stat monitor("Currently Processing:", #ListIndex)
ui drop down("Thread Count:", "2,3,5,10,15,20,30", #threadCount)
set(#threadUsed, 0, "Global")
preloopSetup()
loop(#ListMax) {
    loop while($comparison(#threadUsed, ">=", #threadCount)) {
        wait(1)
    }
    loopProcess()
}
alert("Success! {$list total(%work_list)} Websites Scanned")
alert("Don\'t Forget to Save the CSV File")

TAB 2 CODE:

define initialSetup {
    clear list(%work_list)
    clear table(&url_table)
    clear cookies
    allow images("No")
    allow popups("No")
    allow flash("No")
    allow javascript("No")
    allow css("No")
}
define preloopSetup {
    create table from file(#list, &url_table)
    add list to list(%work_list, $list from file(#list), "Don\'t Delete", "Global")
    set(#ListMax, $list total(%work_list), "Global")
    decrement(#ListMax)
    set(#ListIndex, 0, "Global")
}
define loopProcess {
    increment(#threadUsed)
    increment(#ListIndex)
    thread {
        in new browser {
            mainProcess()
            decrement(#threadUsed)
        }
    }
}
define mainProcess {
    loop while($comparison(#ListIndex, "<=", #ListMax)) {
        then {
            comment("CODE TO BE MULTI-THREADED HERE")
            set(#URLPosition, $list item(%work_list, #ListIndex), "Global")
            navigate(#URLPosition, "Wait")
            wait for browser event("Page Loaded", 5)
            wait(1.5)
            if($search page("keyword")) {
                then {
                    set table cell(&url_table, #ListIndex, 1, "\"Keyword\" Found")
                }
                else {
                    set table cell(&url_table, #ListIndex, 1, "NONE")
                }
            }
            clear cookies
            close page
            set user agent("Internet Explorer 6")
        }
        else {
        }
    }
}

List of random URLs (to save as CSV for the bot):

 

https://pastebin.com/45s5hLZS

Link to post
Share on other sites

The code still needs some work but I modified the multithreading part and that should now work a lot better for you. Basically you want to pass things like list items or when you increment a variable these things should be modified outside of the threads and then passed into them. Otherwise you get 2 threads doing the same thing and it doesn't work right.

on load("Bot Loaded") {
    navigate("about:blank","Wait")
    alert("Please Choose a CSV File to Process")
}
initialSetup()
ui open file("CSV List To Process:",#list)
ui button("Click to Save Updated CSV File") {
    save to file(#list,&url_table)
}
ui stat monitor("Total Pages to Scan:",#ListMax)
ui stat monitor("Currently Processing:",#ListIndex)
ui drop down("Thread Count:","2,3,5,10,15,20,30",#threadCount)
set(#threadUsed,0,"Global")
preloopSetup()
loop(#ListMax) {
    loop while($comparison(#threadUsed,">=",#threadCount)) {
        wait(1)
    }
    loopProcess($next list item(%work_list))
    wait(0.1)
}
alert("Success! {$list total(%work_list)} Websites Scanned")
alert("Don\'t Forget to Save the CSV File")

And

define initialSetup {
    clear list(%work_list)
    clear table(&url_table)
    clear cookies
    allow images("No")
    allow popups("No")
    allow flash("No")
    allow javascript("No")
    allow css("No")
}
define preloopSetup {
    create table from file(#list,&url_table)
    add list to list(%work_list,$list from file(#list),"Don\'t Delete","Global")
    set(#ListMax,$list total(%work_list),"Global")
    decrement(#ListMax)
    set(#ListIndex,0,"Global")
}
define loopProcess(#url) {
    increment(#threadUsed)
    increment(#ListIndex)
    thread {
        in new browser {
            mainProcess(#url, #ListIndex)
        }
        decrement(#threadUsed)
    }
}
define mainProcess(#url, #index) {
    comment("CODE TO BE MULTI-THREADED HERE")
    navigate(#url,"Wait")
    wait for browser event("Page Loaded",5)
    wait(1.5)
    if($search page("keyword")) {
        then {
            set table cell(&url_table,#index,1,"\"Keyword\" Found")
        }
        else {
            set table cell(&url_table,#index,1,"NONE")
        }
    }
    clear cookies
    close page
    set user agent("Internet Explorer 6")
}
  • Like 1
Link to post
Share on other sites

Hey, thanks man I appreciate the help!

 

That got it running and it somewhat works. It skips the first 5 list items for some reason (looking into it). 

 

My main concern now is that I'm getting browser crashing / errors:

 

Problem signature:
  Problem Event Name: BEX
  Application Name: Browser.exe
  Application Version: 0.0.0.0
  Application Timestamp: 510c4659
  Fault Module Name: StackHash_5861
  Fault Module Version: 0.0.0.0
  Fault Module Timestamp: 00000000
  Exception Offset: PCH_AA_FROM_ntdll+0x0003C8AC
  Exception Code: c0000005
  Exception Data: 00000008
  OS Version: 6.3.9600.2.0.0.768.101
  Locale ID: 4105
  Additional Information 1: 5861
  Additional Information 2: 5861822e1919d7c014bbb064c64908b2
  Additional Information 3: 5f25
  Additional Information 4: 5f2531ae070278f893fa99352dadd49e
 
Almost all the threads seem to be frozen or broken or something after a short time.. I took a quick vid: 
 
 
It takes the Ubot browser around 10 seconds to open each website even in the single threaded version, but in my normal browser outside of Ubot it takes 0.5 seconds. It seems like something's very wrong. 
 
Any idea what could cause these issues?
Link to post
Share on other sites

It's hard to say, it looks like you're using Ubot 4 and so depending on what version of Ubot you're using there could be different issues. Also, since you're using V4 the mulithtreaded code needs to change and you need to use the threads counter plugin because of this: http://network.ubotstudio.com/forum/index.php?/topic/15122-must-read-threading-doesnt-work-as-expected-tested-in-v4/

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...