[solved]Scraping Twitter Times

[katelynnm 1]

I have a bot that needs to scrape the seconds (#s), minutes (#m) or hours (#h) that is listed on the first 15 search results.  It just won't save the times properly at all-I can't get them separated into separate rows or do a replace to add in commas or anything.  I'm very stuck-what am I doing wrong?

 

Here is my code:

clear list(%search term)
clear list(%times)
navigate("https://twitter.com/search-home", "Wait")
type text(<id="search-home-input">, "puppy", "Standard")
click(<class="button btn primary-btn submit selected search-btn">, "Left Click", "No")
wait for element(<title="Find the most recent tweets">, "", "Appear")
click(<title="Find the most recent tweets">, "Left Click", "No")
wait for element(<title="Find the most recent tweets">, "", "Appear")
set(#time, $scrape attribute(<span,class="_timestamp js-short-timestamp js-relative-timestamp">, "innerhtml"), "Global")
set(#seconds, $replace(%times, "s", "s, {$new line}"), "Global")
set(#minutes, $replace(#time, "m", "m, {$new line}"), "Global")
add item to list(%times, #time, "Don\'t Delete", "Global")
save to file("C:\\Users\\Katelynn\\Desktop\\times.csv", %times)
set(#term, $scrape attribute(<class="search-query">, "innertext"), "Global")
loop(15) {
    add item to list(%search term, #term, "Don\'t Delete", "Global")
}
save to file("C:\\Users\\Katelynn\\Desktop\\search.txt", %search term)

Edited October 30, 2013 by katelynnm

[Steve 30]

Not exactly sure if I understand what you are trying to accomplish, but if you are trying to get all of the times into their own row of a list, I made a few changes to your code:

clear list(%search term)
clear list(%times)
navigate("https://twitter.com/search-home", "Wait")
type text(<id="search-home-input">, "puppy", "Standard")
click(<class="button btn primary-btn submit selected search-btn">, "Left Click", "No")
wait for element(<title="Find the most recent tweets">, "", "Appear")
click(<title="Find the most recent tweets">, "Left Click", "No")
wait for element(<title="Find the most recent tweets">, "", "Appear")
add list to list(%times, $scrape attribute(<data-time=w"*">, "innertext"), "Delete", "Global")
save to file("{$special folder("Desktop")}\\times.csv", %times)
set(#term, $scrape attribute(<class="search-query">, "innertext"), "Global")
loop(15) {
    add item to list(%search term, #term, "Don\'t Delete", "Global")
}
save to file("{$special folder("Desktop")}\\search.txt", %search term)

[katelynnm 1]

Hi Steve-that works great!  Is there any way that I can now set a limit so that it only scrapes the first 15 times?

[Steve 30]

Hi Steve-that works great!  Is there any way that I can now set a limit so that it only scrapes the first 15 times?

 

It looked like by default it was already doing that? It seemed like it was only loading 15 results unless you scroll down on the page after it loads? 

If you need to make sure its only the first 15, the code will need to be modified, and it will prob. require the use offsets and adding each item to the list individually.

[katelynnm 1]

Thanks again, Steve-it's working well!

[Steve 30]

Thanks again, Steve-it's working well!

 

Glad to help.