Jump to content



Denethor

Member Since 01 Aug 2013
Offline Last Active May 23 2019 02:18 PM
-----

Posts I've Made

In Topic: Any Way To Detect If All Threads Are Finished Before Proceeding?

25 July 2018 - 08:25 AM

I added a variable to the script and track it myself, but it seems there should be a way in Ubot to track this.

It increments the variable in the loop that spawns the thread and then the last line of the defined thread is to decrement that same variable.

Then I added a new loop in my code that just waits until the variable =0 before continuing on.

Crude, but it works


In Topic: Save Downloaded File Very Slow When Threaded

06 June 2018 - 09:46 AM

yes, as I stated above the code removed from the thread runs just fine.

yes, I wrote it.

If I remove all the comments and alerts the error goes away.

Here is the current code, the only change is I moved the saving of the log file to outside the main loop since there is no need writing it unless the loop completed.

I changed all my pathing to C: for this example, that is not my real path, I don't save to the root of my drive :)

I also included all the non threaded parts here now.

clear cookies
create table from file("C:\\SpecLog.txt",&Login)
set(#StepLog,0,"Global")
set(#StepNew,0,"Global")
loop while($table cell(&Login,#StepLog,0) != "") {
    if($table cell(&Login,#StepLog,5) = "false") {
        then {
            add list to table as row(&NEW,#StepNew,0,$plugin function("TableCommands.dll", "$list from table", &Login, "Row", #StepLog))
            set table cell(&NEW,#StepNew,7,#StepLog)
            increment(#StepNew)
        }
        else {
        }
    }
    increment(#StepLog)
}
add list to table as row(&NEW,#StepNew,0,$plugin function("TableCommands.dll", "$list from table", &Login, "Row", #StepLog))
set(#TotalRecords,$subtract($table total rows(&NEW),1),"Global")
set(#WorkToDo,$plugin function("TableCommands.dll", "$table search", &NEW, "false", "Row Index"),"Global")
if(#WorkToDo < 0) {
    then {
        stop script
    }
    else {
    }
}
if($file exists("C:\\StatsResults.zip")) {
    then {
        delete file("C:\\StatsResults.zip")
    }
    else {
    }
}
set(#DBbackup,"C:\\{$random text(8)}.gzip","Global")
set(#LoginCount,0,"Global")
set(#MainLoop,0,"Global")
create table from file("C:\\Empty.csv",&EmptyFile)
set(#a,0,"Global")
loop while($table cell(&NEW,#a,0) != "") {
    loop(2) {
        MyLogin($table cell(&NEW,#a,0), $table cell(&NEW,#a,1), $table cell(&NEW,#a,2), $table cell(&NEW,#a,3), $table cell(&NEW,#a,4), $table cell(&NEW,#a,7))
        increment(#a)
    }
    wait(60)
}
define MyLogin(#id, #pass, #City, #CityURL, #Expected, #LoginIndex) {
    thread {
        set(#CSVfilename,$add($add("C:\\temp\\temp",#City),".csv"),"Local")
        set(#ResultFile,$add($add("C:\\temp\\Result",#City),".jpg"),"Local")
        in new browser {
            navigate(#CityURL,"Wait")
            wait for element(<login link>,"","Appear")
            wait for element($element offset(<name="remember_me">,1),"","Appear")
            change checkbox($element offset(<name="remember_me">,1),"Unchecked")
            type text($element offset(<email field>,1),#id,"Standard")
            wait(2)
            type text($element offset(<password field>,1),#pass,"Standard")
            wait(2)
            click($element offset(<login button>,1),"Left Click","No")
            wait for element(<class="analysis-body">,"","Appear")
            navigate(#CityURL,"Wait")
            wait(5)
            set(#FileSave,0,"Local")
            set(#MoreTime,0,"Local")
            loop while(#FileSave = 0) {
                set(#ErrorSeen,$scrape attribute($element offset(<tagname="h1">,0),"innertext"),"Local")
                if(#ErrorSeen = "Log in") {
                    then {
                        set(#FileSave,1,"Local")
                    }
                    else {
                    }
                }
                set(#ErrorSeen,$url,"Local")
                if(#ErrorSeen = "https://mysite.com/about") {
                    then {
                        set(#FileSave,1,"Local")
                    }
                    else {
                    }
                }
                if(#FileSave = 0) {
                    then {
                        wait for element(<class="Icon Icon--featherDownload">,"","Appear")
                        click(<class="Icon Icon--caretDown controls-caret">,"Left Click","No")
                        click(<innertext="Last 7 Days">,"Left Click","No")
                        wait(5)
                        click(<class="btn btn-default ladda-button">,"Left Click","No")
                        wait(2)
                        save downloaded file(#CSVfilename)
                        set(#MoreTime,0,"Local")
                        loop while(#MoreTime = 0) {
                            set(#ButtonState,$scrape attribute(<id="export">,"outerhtml"),"Local")
                            if($find index(#ButtonState,"data-style=\"expand-right\" disabled=\"\" data-loading=\"\">") < 0) {
                                wait(7)
                                then {
                                    comment("Button State changed back and file is supposedly Saved")
                                    if($file exists(#CSVfilename)) {
                                        then {
                                            set(#FileSize,$plugin function("Advanced Shell.dll", "$shell batch hidden", "for %I in ({#CSVfilename}) do @echo.%~zI"),"Local")
                                            if(#FileSize = 0) {
                                                then {
                                                    delete file(#CSVfilename)
                                                    set(#MoreTime,1,"Local")
                                                }
                                                else {
                                                    clear table(&CSVfilename)
                                                    create table from file(#CSVfilename,&CSVfilename)
                                                    set(#Actual,$table cell(&CSVfilename,1,1),"Local")
                                                    set(#IndexFound,$find index(#Actual,#Expected),"Local")
                                                    if($comparison(&CSVfilename,"= Equals",&EmptyFile)) {
                                                        then {
                                                            set table cell(&Login,#LoginIndex,5,"true")
                                                            set table cell(&Login,#LoginIndex,6,$now)
                                                            set(#FileSave,1,"Local")
                                                            set(#MoreTime,1,"Local")
                                                        }
                                                        else {
                                                            if($find index(#Actual,#Expected) = 0) {
                                                                then {
                                                                    set table cell(&Login,#LoginIndex,5,"true")
                                                                    set table cell(&Login,#LoginIndex,6,$now)
                                                                    set(#FileSave,1,"Local")
                                                                    set(#MoreTime,1,"Local")
                                                                }
                                                                else {
                                                                    set table cell(&Login,#LoginIndex,5,"false")
                                                                    set table cell(&Login,#LoginIndex,6,$now)
                                                                    delete file(#CSVfilename)
                                                                    set(#MoreTime,1,"Local")
                                                                }
                                                            }
                                                        }
                                                    }
                                                }
                                            }
                                        }
                                        else {
                                            set(#MoreTime,1,"Local")
                                        }
                                    }
                                }
                                else {
                                    comment("Data Loading")
                                    wait(4)
                                }
                            }
                        }
                    }
                }
            }
            wait($rand(1,10))
            save to file("C:\\SpecLog.txt",&Login)
        }
    }
}
comment("ADD DB Import here")

The wait(7) before the 'comment("Button State changed back and file is supposedly Saved")' is what had to be added in order for the thread to see the file had been saved, that is not needed when it's not threaded. Without the pause, it always failed with file doesn't exist.

I also added the random wait near the end when it writes the table back out to the log file, because I had an issue where two threads finished at the same time and both tried to write to the file at the same time. Is there some way to check if the file is writable before saving? So the script won't error out? Ubot really needs some 'OnError' handling, so I can determine how to recover from some errors without crashing the script.

 

spec log.txt is a simple csv file like this:

UserName , PassWord , SiteName , SiteURL , DownloadIdentifierCode , Completed(true or false) , TimeStamp of last completion

,,,,,, One empty row at the end to detect end of what I want to process, I can move this in the list if during testing I only want to test up to this point without deleting other entries from the list.

In Topic: Tables And Lists In A Thread Local Or Global?

05 June 2018 - 08:37 AM

Hi Denethor,

 

Pash is correct.  However, you can use List and construct each list item as a CSV Row of data.

 

Then at some point in your main bot you can reload the entire List as a Table.

 

I hope that helps you.

 

Buddy

So lists are local and tables are global?


In Topic: Help With Picking Random List Item By Last Time Used

24 May 2018 - 11:13 AM

the line:

set(#TotalRows,$subtract($table total rows(&myTable),1),"Global")

Fixes a problem I had with the $rand command.

I originally had set the index to be simply a random number between 0 and $table total rows(&myTable)

That however does not work because $table total rows(&myTable) does not account for tables starting at position 0, so it would randomly create an empty data set at the end of the file with nothing but the true,$now

 

The rest should be pretty self explanatory, if anyone has questions, feel free to ask.


In Topic: Help With Picking Random List Item By Last Time Used

24 May 2018 - 10:58 AM

Here is the solution I came up with. When "WorkToDo" can't find any records that haven't been used,  For this example I just had it end the script, in my actual script I have it reset all records "Used" flag (Table column 2) to false, so next time it runs the whole list is available again.

&myTable contains 3 columns for this example.

0=Name,1=UsedFlag(true or false),2=TimeStamp of last use

You'll have to make you sample csv if want to test it out.

Something like this:

one,false,05/24/2018 13:27:29

two,false,05/24/2018 11:50:36
three,true,05/24/2018 13:31:54
four,false,05/24/2018 11:47:58
clear table(&myTable)
create table from file("{$special folder("Application")}\\myTable.csv",&myTable)
clear list(%random cells)
set(#TotalRows,$subtract($table total rows(&myTable),1),"Global")
set(#WorkToDo,$plugin function("TableCommands.dll", "$table search", &myTable, "false", "Row Index"),"Global")
if(#WorkToDo < 0) {
    then {
        stop script
    }
}
set(#LoopCount,0,"Global")
set(#ExitLoop,0,"Global")
loop while(#ExitLoop = 0) {
    set(#myTable Index,$rand(0,#TotalRows),"Global")
    set(#datediff,$n ago($n seconds(12,"Hours")),"Global")
    if($either($comparison($table cell(&myTable,#myTable Index,2),"= Equals",$nothing),$comparison($table cell(&myTable,#myTable Index,2),"<",#datediff))) {
        then {
            add item to list(%random cells,#myTable Index,"Don\'t Delete","Global")
            set table cell(&myTable,$random list item(%random cells),2,$now)
            set table cell(&myTable,$random list item(%random cells),1,"true")
            set(#ExitLoop,1,"Global")
        }
    }
    increment(#LoopCount)
    if(#LoopCount > #TotalRows) {
        then {
            set(#LoopCount,0,"Global")
        }
    }
}
save to file("{$special folder("Application")}\\myTable.csv",&myTable)