Jump to content



Photo

Threading... Why Won't It Work?

threading multithreading in new browser thread

  • Please log in to reply
7 replies to this topic

#1 jomcat

jomcat

    Newbie

  • Members
  • Pip
  • 5 posts
  • OS:Windows 10
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Professional Edition

Posted 08 August 2016 - 08:32 AM

Okay so I've read all the posts about threads and how to get them to work but the little blighters just won't work for me. In random threads the vars, lists or tables would be blank and/or store blank data to the DB. I have a bot that follows the following flow:

  • I set the bot to run a schedule that runs a define
  • The define connects to a sqlite DB and gets a list of tasks to perform
  • Each task is a define that contains a thread in a new browser
  • Each define contains var, lists and tables

So the multi-threading veterans out there are thinking... Duh, the lists and tables are not thread safe and are getting overwritten if the define gets run more than once at the same time. I did consider this, so I moved all the list and table work into a thread safe container provided by the TinyMCE (Open.Framework.dll).

 

Sadly this didn't work either, threads continued to fail due to blank vars, lists or tables. So I decided to make my own version of threading, I set a global variable for each thread to say if it was in use or not. That way a specific define could only be run once at one time. You guessed it... It didn't work. Over the space of time the bot would end out not running any tasks because it wouldn't 'unlock' the current define for some reason.

 

Any help would be great, also how does the Scheduler impact all this? Say the scheduler runs the letsgo() define every 3 minutes and the letsgo() define runs another define that contains a thread that takes 10 minutes to run. I presume the threadded define continues to run regardless of the scheduler kicking off the letsgo() define again?

 

Cheers!



#2 jomcat

jomcat

    Newbie

  • Members
  • Pip
  • 5 posts
  • OS:Windows 10
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Professional Edition

Posted 08 August 2016 - 10:03 AM

Just an update on why the TinyMCE threading plugin didn't work... It doesn't work with "In new browser" http://network.ubots...ripts/?p=108623.

So I guess my long post can now be boiled down to one question... How do you guys make thread safe bots when a define with new thread contains multiple lists, tables and SQLite queries?

 

Cheers!



#3 HelloInsomnia

HelloInsomnia

    Advanced Member

  • Moderators
  • 2139 posts
  • OS:Windows 10
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 10 August 2016 - 10:00 AM

Keep vars and lists local and tables should be okay to work with even though they are global. Store info in a table and at the end shoot it over to the db. The thread should run out - as far as I know - I don't think the scheduler will kill it and it's easy to setup a test real quick to prove that (or disprove it).



#4 jomcat

jomcat

    Newbie

  • Members
  • Pip
  • 5 posts
  • OS:Windows 10
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Professional Edition

Posted 10 August 2016 - 11:57 AM

Hello HelloInsomnia,

 

Thanks for getting back to me. I had already started building a testing bot and found a lot out from that. I do keep all vars and lists local but when the same define (containing a thread) is run more than once at the same time the threads end out storing nothing to the DB or just fails because some bit of data is missing (URL to navigate to etc.) that came from the DB.

 

If you say that local vars and local lists are thread save then my issue must be with DB queries with results overwriting the Global tables from other threads. I have where possible moved this data into local lists asap but still get the same issues arising.

 

I will continue to test, test and test but thank you for your insights.



#5 HelloInsomnia

HelloInsomnia

    Advanced Member

  • Moderators
  • 2139 posts
  • OS:Windows 10
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 10 August 2016 - 01:00 PM

They are thread safe but keep in mind they are not local to the thread but rather the define (custom command or function) they are in. That may be part of your problem with the missing data. Also, you may want to communicate with a table and then push the info to the db at the end without multithreading.



#6 jomcat

jomcat

    Newbie

  • Members
  • Pip
  • 5 posts
  • OS:Windows 10
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Professional Edition

Posted 11 August 2016 - 02:36 AM

Now that's something I can't get my head around... Local to the define but not to the thread? So if I run a define twice and the define contains all its code in a new thread (in new browser) surely the threads that are running will populate the local vars and lists that are in the define the thread is running in? Sorry, I'm sure you've gone through this a thousand times before but I know if I could just get my head around this concept things would click for me (pardon the pun). Also if I store data in a table during the thread and then save it to DB at the end outside of the thread will the data not get messed up during the thread if more than one define/thread is running and also storing data to that global table?

 

I've included some sample code of how I'm structuring my bot in the hope it will help. I will continue to test as that seems to be the key and thanks for all your help with this!

comment("Run ScheduleLoop using Ubot scheduler")
define ScheduleLoop {
	comment("Get list of tasks from DB into &AllTasks. Each row in the table contains a task to run")
	comment("The table &AllTasks could contain up to 20 rows.")
	set(#LoopCounter,0,"Local")
    loop($table total rows(&AllTasks)) {
        if($comparison($table cell(&AllTasks, #LoopCounter, 0),"= Equal","TaskOne")) {
            then {
                TaskOne()
                wait(0.25)
            }
            else if($comparison($table cell(&AllTasks, #LoopCounter, 0),"= Equal","TaskTwo")) {
                TaskTwo()
                wait(0.25)
            }
            else if($comparison($table cell(&AllTasks, #LoopCounter, 0),"= Equal","TaskThree")) {
				TaskThree()
                wait(0.25)
            }
        }
        increment(#LoopCounter)
    }       
}
define TaskOne {
	thread {
        in new browser {
			comment("Run code here with vars and lists Local but DB queries with results still stored in Global table")
			close page
		}
	}
}
define TaskTwo {
	thread {
        in new browser {
			comment("Run code here with vars and lists Local but DB queries with results still stored in Global table")
			close page
		}
	}
}
define TaskThree {
	thread {
        in new browser {
			comment("Run code here with vars and lists Local but DB queries with results still stored in Global table")
			close page
		}
	}
}


#7 jomcat

jomcat

    Newbie

  • Members
  • Pip
  • 5 posts
  • OS:Windows 10
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Professional Edition

Posted 12 August 2016 - 01:17 AM

HelloInsomnia I just want to thank you for your help. It turns out that my issue isn't with threading but with multiple threads trying to access the same db at the same time (as you suggested). As a solution I wrote up some code to queue all database queries to prevent two threads trying to access the same DB. So far so good, running for a few hours with no errors. For those who might have the same issue I've included the code below. It's easy use and I hope it will help somebody who was having the same database threading issues I was having!

comment("USAGE: When making a DB call add the if statement below and include your DB code in the then")
if($comparison($DB_LOCK(),"!= Does not equal",0)) {
	then {
		comment("ADD DB CODE HERE")
		comment("ADD CODE HERE TO MOVE DATA FROM TABLE RESULTS TO LOCAL LISTS")
		DB_UNLOCK()
	}
	else {
		alert("Put your own error handling here")
		return(0)
	}
}
comment("USAGE: Add $DB_LOCK and DB_UNLOCK below just once within your code")
define $DB_LOCK {
    set(#dbLockName,"TASK{$rand(1000,9999)}{$rand(1000,9999)}{$rand(1000,9999)}","Local")
    add item to list(%dbLockList,#dbLockName,"Don't Delete","Global")
    set(#dbLockCounter,0,"Local")
    loop while($comparison($list item(%dbLockList,0),"!= Does not equal",#dbLockName)) {
        wait(5)
        increment(#dbLockCounter)
        if($comparison(#dbLockCounter,"> Greater than",30)) {
            then {
                return(0)
            }
        }
    }
    return(1)
}
define DB_UNLOCK {
    remove from list(%dbLockList,0)
}


#8 HelloInsomnia

HelloInsomnia

    Advanced Member

  • Moderators
  • 2139 posts
  • OS:Windows 10
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 13 August 2016 - 08:30 AM

Jomcat check PM







Also tagged with one or more of these keywords: threading, multithreading, in new browser, thread

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users