Amazon suggest scraping -- how to?

[Joe118 3]

I'd love to see a sample of code that will let me scrape the suggestions that amazon displays when you enter some word. Right now my problem is that I cannot for the life of me get the HTML for the drop down menu that Amazon displays.

 

I try the following:

 

nav to http://www.amazon.com

choose by attribute name field-keywords exact-match

change chosen attribute value "bread machine" (as an example)

send keys chosen $special keys: down

delay 1

 

OK and now what... Thanks for any ideas. Aaron Nimocks or anyone else from the ubot guru cadre? PLEASE?

[greencat 18]

I'd love to see a sample of code that will let me scrape the suggestions that amazon displays when you enter some word. Right now my problem is that I cannot for the life of me get the HTML for the drop down menu that Amazon displays.

 

I try the following:

 

nav to http://www.amazon.com

choose by attribute name field-keywords exact-match

change chosen attribute value "bread machine" (as an example)

send keys chosen $special keys: down

delay 1

 

OK and now what... Thanks for any ideas. Aaron Nimocks or anyone else from the ubot guru cadre? PLEASE?

 

Grab yourself a copy of Firefox and Firebug and go a browsing to Amazon and do a bit of investigating. Turn on inspect element. This is a great tool for finding those tricky elements that dynamically appear.

 

I suspect the divs the search words are on is dynamically generated by JavaScript. The enclosing div's id is srch_sggst. I reckon you should be able to scrape the contents of this layer as innertext(although you'll need to junk the first line as that just says Search suggestions).

 

Update: I had a quick go at this and used JavaScript to scrape the innertext of srch_sggst to a file. See attached.

grab_amazon_suggest.ubot

[Frank 177]

Good suggestion. Firebug is your friend. You'll need to take some time to understand it, but it will help you greatly. Also, pull up the page in internet explorer and look at the page source. Use the search feature to find text around the problem child and see what you can see.

 

Also, don't be afraid to use the different identifiers you can access in the 'choose attributes' to select <a> tags, <span> tags and the such. Your answer at times comes with a bit of hunting!

 

I took a peek myself and you are definitely going to need to hunt the code on the page to see what javascript is being run when that search field is changed. Trigger the javascript code within ubot and then you should be able to see the <div> that is lit up and you can then scrape the div.

 

I won't lie to you, you've got your work cut out for yourself.

 

 

Frank

[Joe118 3]

Greencat and Frank, thanks guys! You both saved the day. I will use your code in the product, Greencat, and you'll get a free copy for your contribution.

 

And Greencat, you taught me another trick here -- I didnt know you could use eval to run javascript code and capture the value, like you do -- VERY cool.

[greencat 18]

Greencat and Frank, thanks guys! You both saved the day. I will use your code in the product, Greencat, and you'll get a free copy for your contribution.

 

And Greencat, you taught me another trick here -- I didnt know you could use eval to run javascript code and capture the value, like you do -- VERY cool.

 

That's excellent Joe118. Look forward to seeing the finished bot. Eval and Javascript combined are extremely powerful. I've used it to solve reliability problems in the past instead of using Ubot's tools to select and scrape values (you can even build in error handling). If you need to scrape or change values 1000s of times a day - javascript has the edge over Ubot's built in tools. Good luck!

[panoz 1]

this javascript code is buggy, it works for the first few loops in the bot then it messes up the entire bot, keeps looping through the same word.

[JohnB 255]

There is no trick to scraping Amazon suggestions. It is actually just straightforward code:

 

ui text box("Keyword:", #kw)

ui stat monitor("Collected: ", #collected)

define Amazon {

navigate("http://www.amazon.com/", "Wait")

type text(<name="field-keywords">, #kw, "Standard")

wait(3)

add list to list(%suggestions, $scrape attribute(<class="suggest_link">, "outertext"), "Delete", "Global")

set(#collected, $list total(%suggestions), "Global")

}

Amazon()

 

 

John

[yodayick 2]

this is a good example. thanx

[Legend 181]

Excellent example... thanks again John!!

 

Just out of curiosity, how would you go about looping this, i.e., going x levels deep? Is there an easy way? I considered a dropdown version, levels 1-10 with separate code for each level, but I thought there must be an easier way.

[Legend 181]

nevermind... I got it... http://ubotstudio.com/forum/public/style_emoticons/default/smile.gif