botmaker7 5 Posted November 15, 2017 Report Share Posted November 15, 2017 The bot loops through a list of URLs and searches each site for a keyword. Then it saves the results in the CSV. Any idea what I'm doing incorrectly? The bot skips some of the first list items, and each thread loads the same URL.. I tried fixing it for hours, but no luck. TAB 1 CODE: on load("Bot Loaded") { navigate("about:blank", "Wait") alert("Please Choose a CSV File to Process") } initialSetup() ui open file("CSV List To Process:", #list) ui button("Click to Save Updated CSV File") { save to file(#list, &url_table) } ui stat monitor("Total Pages to Scan:", #ListMax) ui stat monitor("Currently Processing:", #ListIndex) ui drop down("Thread Count:", "2,3,5,10,15,20,30", #threadCount) set(#threadUsed, 0, "Global") preloopSetup() loop(#ListMax) { loop while($comparison(#threadUsed, ">=", #threadCount)) { wait(1) } loopProcess() } alert("Success! {$list total(%work_list)} Websites Scanned") alert("Don\'t Forget to Save the CSV File")TAB 2 CODE: define initialSetup { clear list(%work_list) clear table(&url_table) clear cookies allow images("No") allow popups("No") allow flash("No") allow javascript("No") allow css("No") } define preloopSetup { create table from file(#list, &url_table) add list to list(%work_list, $list from file(#list), "Don\'t Delete", "Global") set(#ListMax, $list total(%work_list), "Global") decrement(#ListMax) set(#ListIndex, 0, "Global") } define loopProcess { increment(#threadUsed) increment(#ListIndex) thread { in new browser { mainProcess() decrement(#threadUsed) } } } define mainProcess { loop while($comparison(#ListIndex, "<=", #ListMax)) { then { comment("CODE TO BE MULTI-THREADED HERE") set(#URLPosition, $list item(%work_list, #ListIndex), "Global") navigate(#URLPosition, "Wait") wait for browser event("Page Loaded", 5) wait(1.5) if($search page("keyword")) { then { set table cell(&url_table, #ListIndex, 1, "\"Keyword\" Found") } else { set table cell(&url_table, #ListIndex, 1, "NONE") } } clear cookies close page set user agent("Internet Explorer 6") } else { } } } List of random URLs (to save as CSV for the bot): https://pastebin.com/45s5hLZS Quote Link to post Share on other sites
HelloInsomnia 1103 Posted November 15, 2017 Report Share Posted November 15, 2017 The code still needs some work but I modified the multithreading part and that should now work a lot better for you. Basically you want to pass things like list items or when you increment a variable these things should be modified outside of the threads and then passed into them. Otherwise you get 2 threads doing the same thing and it doesn't work right. on load("Bot Loaded") { navigate("about:blank","Wait") alert("Please Choose a CSV File to Process") } initialSetup() ui open file("CSV List To Process:",#list) ui button("Click to Save Updated CSV File") { save to file(#list,&url_table) } ui stat monitor("Total Pages to Scan:",#ListMax) ui stat monitor("Currently Processing:",#ListIndex) ui drop down("Thread Count:","2,3,5,10,15,20,30",#threadCount) set(#threadUsed,0,"Global") preloopSetup() loop(#ListMax) { loop while($comparison(#threadUsed,">=",#threadCount)) { wait(1) } loopProcess($next list item(%work_list)) wait(0.1) } alert("Success! {$list total(%work_list)} Websites Scanned") alert("Don\'t Forget to Save the CSV File") And define initialSetup { clear list(%work_list) clear table(&url_table) clear cookies allow images("No") allow popups("No") allow flash("No") allow javascript("No") allow css("No") } define preloopSetup { create table from file(#list,&url_table) add list to list(%work_list,$list from file(#list),"Don\'t Delete","Global") set(#ListMax,$list total(%work_list),"Global") decrement(#ListMax) set(#ListIndex,0,"Global") } define loopProcess(#url) { increment(#threadUsed) increment(#ListIndex) thread { in new browser { mainProcess(#url, #ListIndex) } decrement(#threadUsed) } } define mainProcess(#url, #index) { comment("CODE TO BE MULTI-THREADED HERE") navigate(#url,"Wait") wait for browser event("Page Loaded",5) wait(1.5) if($search page("keyword")) { then { set table cell(&url_table,#index,1,"\"Keyword\" Found") } else { set table cell(&url_table,#index,1,"NONE") } } clear cookies close page set user agent("Internet Explorer 6") } 1 Quote Link to post Share on other sites
botmaker7 5 Posted November 15, 2017 Author Report Share Posted November 15, 2017 Hey, thanks man I appreciate the help! That got it running and it somewhat works. It skips the first 5 list items for some reason (looking into it). My main concern now is that I'm getting browser crashing / errors: Problem signature: Problem Event Name: BEX Application Name: Browser.exe Application Version: 0.0.0.0 Application Timestamp: 510c4659 Fault Module Name: StackHash_5861 Fault Module Version: 0.0.0.0 Fault Module Timestamp: 00000000 Exception Offset: PCH_AA_FROM_ntdll+0x0003C8AC Exception Code: c0000005 Exception Data: 00000008 OS Version: 6.3.9600.2.0.0.768.101 Locale ID: 4105 Additional Information 1: 5861 Additional Information 2: 5861822e1919d7c014bbb064c64908b2 Additional Information 3: 5f25 Additional Information 4: 5f2531ae070278f893fa99352dadd49e Almost all the threads seem to be frozen or broken or something after a short time.. I took a quick vid: https://www.screencast.com/t/APJ8gZ8dtwK5 It takes the Ubot browser around 10 seconds to open each website even in the single threaded version, but in my normal browser outside of Ubot it takes 0.5 seconds. It seems like something's very wrong. Any idea what could cause these issues? Quote Link to post Share on other sites
HelloInsomnia 1103 Posted November 16, 2017 Report Share Posted November 16, 2017 It's hard to say, it looks like you're using Ubot 4 and so depending on what version of Ubot you're using there could be different issues. Also, since you're using V4 the mulithtreaded code needs to change and you need to use the threads counter plugin because of this: http://network.ubotstudio.com/forum/index.php?/topic/15122-must-read-threading-doesnt-work-as-expected-tested-in-v4/ Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.