Jump to content
UBot Underground

Save Downloaded File Very Slow When Threaded


Recommended Posts

Here is a basic script I tried to thread to speed up the time to process all records. It's not resource issue as I can limit it to one thread at a time and it still doesn't work in a timely fashion. The same code, not in a thread, saves instantly. Thread it and it takes 10 to 14 sec for the file to appear in the folder after the file is actually downloaded.

loop while($table cell(&NEW,#a,0) != "") {
    loop(2) {
        MyLogin($table cell(&NEW,#a,0), $table cell(&NEW,#a,1), $table cell(&NEW,#a,2), $table cell(&NEW,#a,3), $table cell(&NEW,#a,4))
        increment(#a)
    }
    wait(240)
}
define MyLogin(#id, #pass, #City, #CityURL, #Expected) {
    thread {
        set(#CSVfilename,$add($add("C:\\temp",#City),".csv"),"Local")
        set(#ResultFile,$add($add("C:\\Result",#City),".jpg"),"Local")
        in new browser {
            navigate(#CityURL,"Wait")
            wait for element(<login link>,"","Appear")
            wait for element($element offset(<name="remember_me">,1),"","Appear")
            change checkbox($element offset(<name="remember_me">,1),"Unchecked")
            type text($element offset(<email field>,1),#id,"Standard")
            wait(2)
            type text($element offset(<password field>,1),#pass,"Standard")
            wait(2)
            click($element offset(<login button>,1),"Left Click","No")
            wait for element(<class="analysis-body">,"","Appear")
            navigate(#CityURL,"Wait")
            wait(5)
            set(#FileSave,0,"Local")
            set(#TryCount,0,"Local")
            set(#MoreTime,0,"Local")
            loop while(#FileSave = 0) {
                set(#ErrorSeen,$scrape attribute($element offset(<tagname="h1">,0),"innertext"),"Local")
                if(#ErrorSeen = "Log in") {
                    then {
                        set(#FileSave,1,"Local")
                    }
                    else {
                    }
                }
                set(#ErrorSeen,$url,"Local")
                if(#ErrorSeen = "https://mysite.com/about") {
                    then {
                        set(#FileSave,1,"Local")
                    }
                    else {
                    }
                }
                if(#FileSave = 0) {
                    then {
                        wait for element(<class="Icon Icon--featherDownload">,"","Appear")
                        click(<class="Icon Icon--caretDown controls-caret">,"Left Click","No")
                        click(<innertext="Last 7 Days">,"Left Click","No")
                        wait(5)
                        click(<class="btn btn-default ladda-button">,"Left Click","No")
                        wait(2)
                        save downloaded file(#CSVfilename)
                        set(#MoreTime,0,"Local")
                        loop while(#MoreTime = 0) {
                            set(#ButtonState,$scrape attribute(<id="export">,"outerhtml"),"Local")
                            if($find index(#ButtonState,"data-style=\"expand-right\" disabled=\"\" data-loading=\"\">") < 0) {
                                then {
                                    comment("Button State changed back and file is supposedly Saved")
                                    if($file exists(#CSVfilename)) {
                                        then {
                                            alert("File exists keep verifing")
                                            set(#FileSize,$plugin function("Advanced Shell.dll", "$shell batch hidden", "for %I in ({#CSVfilename}) do @echo.%~zI"),"Local")
                                            alert(#FileSize)
                                            if(#FileSize = 0) {
                                                then {
                                                    delete file(#CSVfilename)
                                                    set(#MoreTime,1,"Local")
                                                    alert("Delete File, set MoreTime 1")
                                                }
                                                else {
                                                    clear table(&CSVfilename)
                                                    create table from file(#CSVfilename,&CSVfilename)
                                                    set(#Actual,$table cell(&CSVfilename,1,1),"Local")
                                                    set(#IndexFound,$find index(#Actual,#Expected),"Local")
                                                    comment("compare to empty here")
                                                    if($comparison(&CSVfilename,"= Equals",&EmptyFile)) {
                                                        then {
                                                            comment("File was saved and contains nothing but headers, no data")
                                                            set table cell(&Login,$plugin function("TableCommands.dll", "$table search", &Login, #CityURL, "Row Index"),5,"true")
                                                            set table cell(&Login,$plugin function("TableCommands.dll", "$table search", &Login, #CityURL, "Row Index"),6,$now)
                                                            set(#FileSave,1,"Local")
                                                            set(#MoreTime,1,"Local")
                                                        }
                                                        else {
                                                            Comment("File saved contians data continue chekcing")
                                                            if($find index(#Actual,#Expected) = 0) {
                                                                then {
                                                                    comment("File contains data expected, exit filesave loop, exit moretime loop")
                                                                    set table cell(&Login,$plugin function("TableCommands.dll", "$table search", &Login, #CityURL, "Row Index"),5,"true")
                                                                    set table cell(&Login,$plugin function("TableCommands.dll", "$table search", &Login, #CityURL, "Row Index"),6,$now)
                                                                    set(#FileSave,1,"Local")
                                                                    set(#MoreTime,1,"Local")
                                                                }
                                                                else {
                                                                    comment("File contains bad data, delete, end loop, try again")
                                                                    set table cell(&Login,$plugin function("TableCommands.dll", "$table search", &Login, #CityURL, "Row Index"),5,"false")
                                                                    set table cell(&Login,$plugin function("TableCommands.dll", "$table search", &Login, #CityURL, "Row Index"),6,$now)
                                                                    delete file(#CSVfilename)
                                                                    set(#MoreTime,1,"Local")
                                                                    alert("File contains bad data,FS0 MT1, delete file")
                                                                }
                                                            }
                                                        }
                                                        save to file("C:\\SpecLog.txt",&Login)
                                                        alert("Save Login table")
                                                    }
                                                }
                                            }
                                        }
                                        else {
                                            set(#MoreTime,1,"Local")
                                            alert("File Doesn't exist, MT1")
                                        }
                                    }
                                }
                                else {
                                    comment("Data Still Loading")
                                    wait(4)
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

I added all the alerts to see where the thread was failing since you can't see what's going on inside of a thread as far as I know.

It's very basic script, login to site, select time frame, wait for download button to have be in the ready state, click download button, check status of button (when the file is being compiled on the server side the button state changes, so I use that to know when to look for the local file), I pause 4 sec between each check just to not eat up too many resources checking constantly. If the button changes back to ready that signals the file has been sent to the local browser, I detect that and look for the file from the Save Downloaded File command. Threaded like above it always fails to find the file it supposedly saved and tries to download it again, which then fails because the file then already exists. I have left an file explorer window open to see when the file actually gets written out and when not threaded it happens instantly. When threaded, I have to add a 10 or more wait to ensure the file has been saved. 

 

Am I missing something here or is this a bug that needs to be reported?

Edited by Denethor
Link to post
Share on other sites

@Denethor,

 

Well, the code looks nice in that format but it will not load in my system.  There seems to be an error somewhere.

 

Question.

 

Did you develop this code first?  Before you placed it in the Thread?

 

While Alerts can help you, flowing the flow is a much better approach (imho).

 

Buddy

Link to post
Share on other sites

yes, as I stated above the code removed from the thread runs just fine.

yes, I wrote it.

If I remove all the comments and alerts the error goes away.

Here is the current code, the only change is I moved the saving of the log file to outside the main loop since there is no need writing it unless the loop completed.

I changed all my pathing to C: for this example, that is not my real path, I don't save to the root of my drive :)

I also included all the non threaded parts here now.

clear cookies
create table from file("C:\\SpecLog.txt",&Login)
set(#StepLog,0,"Global")
set(#StepNew,0,"Global")
loop while($table cell(&Login,#StepLog,0) != "") {
    if($table cell(&Login,#StepLog,5) = "false") {
        then {
            add list to table as row(&NEW,#StepNew,0,$plugin function("TableCommands.dll", "$list from table", &Login, "Row", #StepLog))
            set table cell(&NEW,#StepNew,7,#StepLog)
            increment(#StepNew)
        }
        else {
        }
    }
    increment(#StepLog)
}
add list to table as row(&NEW,#StepNew,0,$plugin function("TableCommands.dll", "$list from table", &Login, "Row", #StepLog))
set(#TotalRecords,$subtract($table total rows(&NEW),1),"Global")
set(#WorkToDo,$plugin function("TableCommands.dll", "$table search", &NEW, "false", "Row Index"),"Global")
if(#WorkToDo < 0) {
    then {
        stop script
    }
    else {
    }
}
if($file exists("C:\\StatsResults.zip")) {
    then {
        delete file("C:\\StatsResults.zip")
    }
    else {
    }
}
set(#DBbackup,"C:\\{$random text(8)}.gzip","Global")
set(#LoginCount,0,"Global")
set(#MainLoop,0,"Global")
create table from file("C:\\Empty.csv",&EmptyFile)
set(#a,0,"Global")
loop while($table cell(&NEW,#a,0) != "") {
    loop(2) {
        MyLogin($table cell(&NEW,#a,0), $table cell(&NEW,#a,1), $table cell(&NEW,#a,2), $table cell(&NEW,#a,3), $table cell(&NEW,#a,4), $table cell(&NEW,#a,7))
        increment(#a)
    }
    wait(60)
}
define MyLogin(#id, #pass, #City, #CityURL, #Expected, #LoginIndex) {
    thread {
        set(#CSVfilename,$add($add("C:\\temp\\temp",#City),".csv"),"Local")
        set(#ResultFile,$add($add("C:\\temp\\Result",#City),".jpg"),"Local")
        in new browser {
            navigate(#CityURL,"Wait")
            wait for element(<login link>,"","Appear")
            wait for element($element offset(<name="remember_me">,1),"","Appear")
            change checkbox($element offset(<name="remember_me">,1),"Unchecked")
            type text($element offset(<email field>,1),#id,"Standard")
            wait(2)
            type text($element offset(<password field>,1),#pass,"Standard")
            wait(2)
            click($element offset(<login button>,1),"Left Click","No")
            wait for element(<class="analysis-body">,"","Appear")
            navigate(#CityURL,"Wait")
            wait(5)
            set(#FileSave,0,"Local")
            set(#MoreTime,0,"Local")
            loop while(#FileSave = 0) {
                set(#ErrorSeen,$scrape attribute($element offset(<tagname="h1">,0),"innertext"),"Local")
                if(#ErrorSeen = "Log in") {
                    then {
                        set(#FileSave,1,"Local")
                    }
                    else {
                    }
                }
                set(#ErrorSeen,$url,"Local")
                if(#ErrorSeen = "https://mysite.com/about") {
                    then {
                        set(#FileSave,1,"Local")
                    }
                    else {
                    }
                }
                if(#FileSave = 0) {
                    then {
                        wait for element(<class="Icon Icon--featherDownload">,"","Appear")
                        click(<class="Icon Icon--caretDown controls-caret">,"Left Click","No")
                        click(<innertext="Last 7 Days">,"Left Click","No")
                        wait(5)
                        click(<class="btn btn-default ladda-button">,"Left Click","No")
                        wait(2)
                        save downloaded file(#CSVfilename)
                        set(#MoreTime,0,"Local")
                        loop while(#MoreTime = 0) {
                            set(#ButtonState,$scrape attribute(<id="export">,"outerhtml"),"Local")
                            if($find index(#ButtonState,"data-style=\"expand-right\" disabled=\"\" data-loading=\"\">") < 0) {
                                wait(7)
                                then {
                                    comment("Button State changed back and file is supposedly Saved")
                                    if($file exists(#CSVfilename)) {
                                        then {
                                            set(#FileSize,$plugin function("Advanced Shell.dll", "$shell batch hidden", "for %I in ({#CSVfilename}) do @echo.%~zI"),"Local")
                                            if(#FileSize = 0) {
                                                then {
                                                    delete file(#CSVfilename)
                                                    set(#MoreTime,1,"Local")
                                                }
                                                else {
                                                    clear table(&CSVfilename)
                                                    create table from file(#CSVfilename,&CSVfilename)
                                                    set(#Actual,$table cell(&CSVfilename,1,1),"Local")
                                                    set(#IndexFound,$find index(#Actual,#Expected),"Local")
                                                    if($comparison(&CSVfilename,"= Equals",&EmptyFile)) {
                                                        then {
                                                            set table cell(&Login,#LoginIndex,5,"true")
                                                            set table cell(&Login,#LoginIndex,6,$now)
                                                            set(#FileSave,1,"Local")
                                                            set(#MoreTime,1,"Local")
                                                        }
                                                        else {
                                                            if($find index(#Actual,#Expected) = 0) {
                                                                then {
                                                                    set table cell(&Login,#LoginIndex,5,"true")
                                                                    set table cell(&Login,#LoginIndex,6,$now)
                                                                    set(#FileSave,1,"Local")
                                                                    set(#MoreTime,1,"Local")
                                                                }
                                                                else {
                                                                    set table cell(&Login,#LoginIndex,5,"false")
                                                                    set table cell(&Login,#LoginIndex,6,$now)
                                                                    delete file(#CSVfilename)
                                                                    set(#MoreTime,1,"Local")
                                                                }
                                                            }
                                                        }
                                                    }
                                                }
                                            }
                                        }
                                        else {
                                            set(#MoreTime,1,"Local")
                                        }
                                    }
                                }
                                else {
                                    comment("Data Loading")
                                    wait(4)
                                }
                            }
                        }
                    }
                }
            }
            wait($rand(1,10))
            save to file("C:\\SpecLog.txt",&Login)
        }
    }
}
comment("ADD DB Import here")

The wait(7) before the 'comment("Button State changed back and file is supposedly Saved")' is what had to be added in order for the thread to see the file had been saved, that is not needed when it's not threaded. Without the pause, it always failed with file doesn't exist.

I also added the random wait near the end when it writes the table back out to the log file, because I had an issue where two threads finished at the same time and both tried to write to the file at the same time. Is there some way to check if the file is writable before saving? So the script won't error out? Ubot really needs some 'OnError' handling, so I can determine how to recover from some errors without crashing the script.

 

spec log.txt is a simple csv file like this:

UserName , PassWord , SiteName , SiteURL , DownloadIdentifierCode , Completed(true or false) , TimeStamp of last completion

,,,,,, One empty row at the end to detect end of what I want to process, I can move this in the list if during testing I only want to test up to this point without deleting other entries from the list.
Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...