[SOLVED] Threading with local variables

[mk21 1]

I've pasted my code below.  It appears to be doing exactly what I ask.  I get a table with 5 rows and 7 columns of data as a result.  However, the data isn't correct.  Can anyone tell me what I'm doing wrong?

 

clear list(%events)
add list to list(%events, $list from text("http://www.stubhub.com/katy-perry-tickets/katy-perry-raleigh-pnc-arena-22-6-2014-4450701/,http://www.stubhub.com/katy-perry-tickets/katy-perry-washington-verizon-center-washington-dc-24-6-2014-4450702/,http://www.stubhub.com/katy-perry-tickets/katy-perry-nashville-bridgestone-arena-27-6-2014-4450703/,http://www.stubhub.com/katy-perry-tickets/katy-perry-brooklyn-barclays-center-24-7-2014-4450746/,http://www.stubhub.com/katy-perry-tickets/katy-perry-pittsburgh-consol-energy-center-22-7-2014-4450713/", ","), "Delete", "Global")
set(#wait, 3, "Global")
loop($list total(%events)) {
    thread {
        in new browser {
            set(#timestamp, $date, "Local")
            set(#url, $next list item(%events), "Local")
            clear cookies
            navigate(#url, "Wait")
            wait for element(<id="ticketsAvailable">, "", "Appear")
            set(#qty, $scrape attribute(<id="ticketsAvailable">, "innertext"), "Local")
            change dropdown(<id="quantitySelect">, 1)
            wait(#wait)
            set(#single, $trim($replace($replace($scrape attribute($element offset($element offset(<class="q">, 0), 0), "innertext"), "Go", $nothing), "$", $nothing)), "Local")
            change dropdown(<id="quantitySelect">, 2)
            wait(#wait)
            set(#double, $trim($replace($replace($scrape attribute($element offset($element offset(<class="q">, 0), 0), "innertext"), "Go", $nothing), "$", $nothing)), "Local")
            change dropdown(<id="quantitySelect">, 3)
            wait(#wait)
            set(#triple, $trim($replace($replace($scrape attribute($element offset($element offset(<class="q">, 0), 0), "innertext"), "Go", $nothing), "$", $nothing)), "Local")
            change dropdown(<id="quantitySelect">, 4)
            wait(#wait)
            set(#quadruple, $trim($replace($replace($scrape attribute($element offset($element offset(<class="q">, 0), 0), "innertext"), "Go", $nothing), "$", $nothing)), "Local")
            wait(#wait)
            clear list(%data)
            add item to list(%data, #timestamp, "Don\'t Delete", "Local")

            add item to list(%data, $url, "Don\'t Delete", "Local")

            add item to list(%data, #qty, "Don\'t Delete", "Local")

            add item to list(%data, #single, "Don\'t Delete", "Local")

            add item to list(%data, #double, "Don\'t Delete", "Local")

            add item to list(%data, #triple, "Don\'t Delete", "Local")

            add item to list(%data, #quadruple, "Don\'t Delete", "Local")

            add list to table as row(&All Events Data, $table total rows(&All Events Data), 0, %data)
        }
    }
}

[blumi40 222]

should work

ui drop down("Threads", "1,2,3,4,5,6,7,8,9,10", #Threads)
clear list(%events)
add list to list(%events, $list from text("http://www.stubhub.c...7-2014-4450713/", ","), "Delete", "Global")
set(#wait, 3, "Global")
set(#MasterLoopAndROW, 0, "Global")
clear table(&All Events Data)
loop($list total(%events)) {
    set(#Threadloop, 0, "Global")
    thread {
        increment(#Threadloop)
        in new browser {
            set(#timestamp, $date, "Local")
            set(#url, $next list item(%events), "Local")
            clear cookies
            navigate(#url, "Wait")
            wait for element(<id="ticketsAvailable">, "", "Appear")
            set(#qty, $scrape attribute(<id="ticketsAvailable">, "innertext"), "Local")
            change dropdown(<id="quantitySelect">, 1)
            wait(#wait)
            set(#single, $trim($replace($replace($scrape attribute($element offset($element offset(<class="q">, 0), 0), "innertext"), "Go", $nothing), "$", $nothing)), "Local")
            change dropdown(<id="quantitySelect">, 2)
            wait(#wait)
            set(#double, $trim($replace($replace($scrape attribute($element offset($element offset(<class="q">, 0), 0), "innertext"), "Go", $nothing), "$", $nothing)), "Local")
            change dropdown(<id="quantitySelect">, 3)
            wait(#wait)
            set(#triple, $trim($replace($replace($scrape attribute($element offset($element offset(<class="q">, 0), 0), "innertext"), "Go", $nothing), "$", $nothing)), "Local")
            change dropdown(<id="quantitySelect">, 4)
            wait(#wait)
            set(#quadruple, $trim($replace($replace($scrape attribute($element offset($element offset(<class="q">, 0), 0), "innertext"), "Go", $nothing), "$", $nothing)), "Local")
            wait(#wait)
            set table cell(&All Events Data, #MasterLoopAndROW, 0, #timestamp)
            set table cell(&All Events Data, #MasterLoopAndROW, 1, $url)
            set table cell(&All Events Data, #MasterLoopAndROW, 2, #qty)
            set table cell(&All Events Data, #MasterLoopAndROW, 3, #single)
            set table cell(&All Events Data, #MasterLoopAndROW, 4, #double)
            set table cell(&All Events Data, #MasterLoopAndROW, 5, #triple)
        }
        close page
    }
    decrement(#Threadloop)
    increment(#MasterLoopAndROW)
    wait(1)
    loop while($comparison(#Threadloop, ">", #Threads)) {
        wait(2)
    }
}

[mk21 1]

@blumi40.  I tried your script, but it didn't seem to work either.  It would only fill in one of the table rows.  Could some try to run my script?  It takes just a couple minutes to run, and you can see what I'm seeing.  I'm starting to think maybe you can't scrape data in thread mode because any data that is scraped is being copied into the table as all the same, even though the numbers should be different from each url.  Is scraping different data in different threads possible?

[blumi40 222]

 set(#Threadloop, 0, "Global")  have to be outside the loop !
ad pls check your $list from text in your event list

 

ui drop down("Threads", "1,2,3,4,5,6,7,8,9,10", #Threads)

add list to list(%events, $list from text("1,2,3,4,5,6,7,8,9", ","), "Delete", "Global")
set(#wait, 3, "Global")
set(#MasterLoopAndROW, 0, "Global")
clear table(&All Events Data)
set(#Threadloop, 0, "Global")
loop($list total(%events)) {
    increment(#Threadloop)
    thread {
        in new browser {
            set(#timestamp, $date, "Local")
            clear cookies
            wait for element(<id="ticketsAvailable">, "", "Appear")
            set(#qty, $scrape attribute(<id="ticketsAvailable">, "innertext"), "Local")
            change dropdown(<id="quantitySelect">, 1)
            wait(#wait)
            set(#single, $trim($replace($replace($scrape attribute($element offset($element offset(<class="q">, 0), 0), "innertext"), "Go", $nothing), "$", $nothing)), "Local")
            change dropdown(<id="quantitySelect">, 2)
            wait(#wait)
            set(#double, $trim($replace($replace($scrape attribute($element offset($element offset(<class="q">, 0), 0), "innertext"), "Go", $nothing), "$", $nothing)), "Local")
            change dropdown(<id="quantitySelect">, 3)
            wait(#wait)
            set(#triple, $trim($replace($replace($scrape attribute($element offset($element offset(<class="q">, 0), 0), "innertext"), "Go", $nothing), "$", $nothing)), "Local")
            change dropdown(<id="quantitySelect">, 4)
            wait(#wait)
            set(#quadruple, $trim($replace($replace($scrape attribute($element offset($element offset(<class="q">, 0), 0), "innertext"), "Go", $nothing), "$", $nothing)), "Local")
            wait(#wait)
            set table cell(&All Events Data, #MasterLoopAndROW, 0, #timestamp)
            set table cell(&All Events Data, #MasterLoopAndROW, 1, $url)
            set table cell(&All Events Data, #MasterLoopAndROW, 2, #qty)
            set table cell(&All Events Data, #MasterLoopAndROW, 3, #single)
            set table cell(&All Events Data, #MasterLoopAndROW, 4, #double)
            set table cell(&All Events Data, #MasterLoopAndROW, 5, #triple)
        }
        decrement(#Threadloop)
        close page
    }
    increment(#MasterLoopAndROW)
    wait(1)
    loop while($comparison(#Threadloop, ">", #Threads)) {
        wait(2)
    }
}

 

an example which works

[mk21 1]

Ok, that last try didn't seem to work for me either.  Here's where I'm at right now and what it's doing:

 

ui drop down("Threads", "1,2,3,4,5,6,7,8,9,10", #Threads)
add list to list(%events, $list from text("http://www.stubhub.com/katy-perry-tickets/katy-perry-raleigh-pnc-arena-22-6-2014-4450701/,http://www.stubhub.com/katy-perry-tickets/katy-perry-washington-verizon-center-washington-dc-24-6-2014-4450702/,http://www.stubhub.com/katy-perry-tickets/katy-perry-nashville-bridgestone-arena-27-6-2014-4450703/,http://www.stubhub.com/katy-perry-tickets/katy-perry-brooklyn-barclays-center-24-7-2014-4450746/,http://www.stubhub.com/katy-perry-tickets/katy-perry-pittsburgh-consol-energy-center-22-7-2014-4450713/", ","), "Delete", "Global")
set(#wait, 3, "Global")
set(#MasterLoopAndROW, 0, "Global")
clear table(&All Events Data)
set(#Threadloop, 0, "Global")
loop($list total(%events)) {
    increment(#Threadloop)
    thread {
        in new browser {
            set(#timestamp, $date, "Local")
            clear cookies
            navigate($next list item(%events), "Wait")
            wait for element(<id="ticketsAvailable">, "", "Appear")
            set(#qty, $scrape attribute(<id="ticketsAvailable">, "innertext"), "Local")
            change dropdown(<id="quantitySelect">, 1)
            wait(#wait)
            set(#single, $trim($replace($replace($scrape attribute($element offset($element offset(<class="q">, 0), 0), "innertext"), "Go", $nothing), "$", $nothing)), "Local")
            change dropdown(<id="quantitySelect">, 2)
            wait(#wait)
            set(#double, $trim($replace($replace($scrape attribute($element offset($element offset(<class="q">, 0), 0), "innertext"), "Go", $nothing), "$", $nothing)), "Local")
            change dropdown(<id="quantitySelect">, 3)
            wait(#wait)
            set(#triple, $trim($replace($replace($scrape attribute($element offset($element offset(<class="q">, 0), 0), "innertext"), "Go", $nothing), "$", $nothing)), "Local")
            change dropdown(<id="quantitySelect">, 4)
            wait(#wait)
            set(#quadruple, $trim($replace($replace($scrape attribute($element offset($element offset(<class="q">, 0), 0), "innertext"), "Go", $nothing), "$", $nothing)), "Local")
            wait(#wait)
            set table cell(&All Events Data, #MasterLoopAndROW, 0, #timestamp)
            set table cell(&All Events Data, #MasterLoopAndROW, 1, $url)
            set table cell(&All Events Data, #MasterLoopAndROW, 2, #qty)
            set table cell(&All Events Data, #MasterLoopAndROW, 3, #single)
            set table cell(&All Events Data, #MasterLoopAndROW, 4, #double)
            set table cell(&All Events Data, #MasterLoopAndROW, 5, #triple)
        }
        set table cell(&All Events Data, #MasterLoopAndROW, 6, #quadruple)
        decrement(#Threadloop)
        close page
    }
    increment(#MasterLoopAndROW)
    wait(1)
    loop while($comparison(#Threadloop, ">=", #Threads)) {
        wait(2)
    }
}

 

 

 

I think the forum truncates my list from text because of the quotation marks, so here's what my list is:

 

http://www.stubhub.com/katy-perry-tickets/katy-perry-raleigh-pnc-arena-22-6-2014-4450701/

http://www.stubhub.com/katy-perry-tickets/katy-perry-washington-verizon-center-washington-dc-24-6-2014-4450702/

http://www.stubhub.com/katy-perry-tickets/katy-perry-nashville-bridgestone-arena-27-6-2014-4450703/

http://www.stubhub.com/katy-perry-tickets/katy-perry-brooklyn-barclays-center-24-7-2014-4450746/

http://www.stubhub.com/katy-perry-tickets/katy-perry-pittsburgh-consol-energy-center-22-7-2014-4450713/

 

If you just click the truncated part in the script, it will open a browser, and you can copy and pasted the script text with the delimiters in it from the url bar.

 

Now if I run this with just 1 loop, it works fairly well, with 1 exception.  Instead of creating a table with 5 rows of data from the 5 links, it creates a table with 6 rows, the first row is completely empty.  I can work around that if necessary with delete table row, but eventually I'd like to put each link's data into a database rather than a table, so if I can't figure out why that empty row is being created, it won't work well.

 

The other problem that exists is if I run the scrip with 2 threads, I 2 rows of empty data, 2 rows of data that is the exact same except for the timestamp and the url, and 1 row of incorrect data and 1 row of correct data.  With 3 threads, you guessed it, 3 rows of empty data, and so on.

 

I'm starting to understand the threading process after working with the last script you posted, but I'm still a little lost.

[mk21 1]

Ok I think I got it done.  If anyone wants help with this, here's a link to another topic addressing this.  My issue was that I wasn't running it out of a define.  Once I did that, all the local variables and lists cooperated.

 

http://www.ubotstudio.com/forum/index.php?/topic/13847-local-lists-multi-threading/