data mining

[Pete 121]

Would it be possible to tap into the search suggestion drop down the big 3 are all using now days? It seems to send each keystroke

[alcr 135]

What big 3? the search suggestion drop can be scraped, yes.

[Pete 121]

What big 3?

 

google yahoo bing

 

The google/bing search suggestion drop down box is not a standard dropdown as you type your search so it sends the keystrokes one by one and updates the suggestions for each keystroke.

 

Type ubot and the suggestions are (ubot, ubot for kids, ubot game,and so on)

However type ubot and just add the character d and get (ubot download, ubot discount, developer)

 

I would like grab all those words

 

Yes you can choose the attribute for the drop down box but how to add it to a list

1. The box will disappear before you add it to a list

2. The box updates on the fly as you type so how to pin point results after you have typed in your keyword and then added a character

[Kreatus (Ubot Ninja) 422]

Use this tool http://tools.seobook.com/general/keyword-information/ you can scrape the keywords there. though its for google suggest only .

[Pete 121]

Thanks for the reply Kreatus

 

But that’s the problem the whole world is using adwords keyword tool and you have little control with it

 

I think alcr reply was incorrect as I believe he did not check the type of dropdown box in question

And was referring to a run of the mill dropdown box

 

Ubot is by designed to manipulate webpage’s but it seems this could be beyond ubot

This is very significant as it would open up a lot more possibilities in data mining

[adamv 0]

Jon Leger is coming out with a keyword tool in a couple of days that will scrape keywords from the suggestions the big three search engines give. I have no idea what the cost will be yet and I would also be interested in learning if ubot is capable of scraping that data.

[Frank 177]

I too would like to learn how to do just this. I cannot seem to find that code that is firing that box. Hints, tips, ideas anyone?

 

Frank

[Joe118 3]

I'm very close to releasing a product based on ubot that does exactly what Zap describes. It will be a WSO in the warrior forum, so you can learn about it there.

[The_Brit 13]

I created a bot this morning to see if it was doable.

 

Anyone want to buy it?

 

Dave

 

EDIT: I can provide a demo of it, restricted to returning only one of the items in the list. I'll have to email it as the compiled version is too big to upload here. Results are in c:\google_search_suggestions.txt.

[Hugo 2]

I too would like to learn how to do just this. I cannot seem to find that code that is firing that box. Hints, tips, ideas anyone?

 

Frank

 

yeah, I don't know nearly enough about javascript to try to decipher through G's source, but experimenting this evening, I was able to get the suggestion box to fire by using "send keys chosen" - which isn't very effective as that command sends every keystroke 3 times, but not by using "send keys field chosen" any idea why the two commands behave so differently?

 

-Hugo

[Hugo 2]

Replying to my own post is weird, but I just tried something that worked, so I figured I'd share:

 

This works on Google - After you "send keys field chosen" (I bet change chosen attribute would work as well but it's untested) to enter the search string you're looking for, "send keys chosen" and set the text to "$special keys: down"

 

Then the query will look like "[search Term]\\" and the resulting suggestion box will give you the same result that you get in a browser using "[search Term] ".

 

Still haven't figured out how to select and scrape the resulting suggestion box. Anyone have any ideas???

 

Thanks,

 

Hugo

[Pete 121]

Sounds to me like you need to know a fair amount of javascript to pull this off

[Hugo 2]

Sounds to me like you need to know a fair amount of javascript to pull this off

 

Nope.

 

Find a working sample attached.

 

I'm sure it's not nearly as functional as what some people here have built, but it works.

 

(Note: Right now it only returns the first ten suggestions. You'll have to modify it and add loops etc. to make it into a viable keyword research tool)

 

Hope this helps

 

-Hugo

 

P.s. I see that there are people selling working bots. Those are likely better than what I've provided.

 

I am simply sharing my bot sample to help people who are looking to create bots for their PERSONAL USE. PLEASE do not take the sample I am providing and use my work to create a commercial bot to sell.

 

Thanks

suggestions-test.ubot

[alcr 135]

Nope.

 

Find a working sample attached.

 

I'm sure it's not nearly as functional as what some people here have built, but it works.

 

(Note: Right now it only returns the first ten suggestions. You'll have to modify it and add loops etc. to make it into a viable keyword research tool)

 

Hope this helps

 

-Hugo

 

P.s. I see that there are people selling working bots. Those are likely better than what I've provided.

 

I am simply sharing my bot sample to help people who are looking to create bots for their PERSONAL USE. PLEASE do not take the sample I am providing and use my work to create a commercial bot to sell.

 

Thanks

Nice share Hugo! That was precisely what I was talking about earlier.

[Pete 121]

Nice one Hugo

 

Simple solution is always the best there was me playing about with JavaScript Visibility

 

And yes it works in a loop

 

 

Thanks for all your work

[Kreatus (Ubot Ninja) 422]

Nice one hugo! Keep it up!

[Frank 177]

Darn! I didn't know about the special keys that you can send, as a keydown! I had all the rest figured out and was stuck trying to get that stupid box to show up. I had resolved that I'd have to pick through the google cryptic js.

 

Thanks very much for that! Helps me in a lot of places.

 

Frank

[Frank 177]

Here's a stronger challenge for you all. Try to scrape the keyword suggestions out of yahoo! The code on the page is obsured and changes so scraping for me has been pretty much impossible so far.

 

But I shall not give up!!

 

Frank

[The_Brit 13]

I dodn't go through the code on the samples above, but you can achieve getting the drop-down to appear in Google by sending a space after your keyword as well. I used this method some time ago to scrape keywords from the suggestion list and as far as I know it still works.

[The_Brit 13]

Here's a stronger challenge for you all. Try to scrape the keyword suggestions out of yahoo! The code on the page is obsured and changes so scraping for me has been pretty much impossible so far.

 

But I shall not give up!!

 

Frank

Did you try page scraping using

 

/STRONG> for the left and </A></LI> for the right and then just combine your search term with a space before the word you scraped? I tried it and it worked for me.

 

Dave

 

EDIT: You need to have LI as the starting attribute for the page scrape and not not A or STRONG which are a couple of the choices I have seen it default to.

[Frank 177]

Did you try page scraping using

 

/STRONG> for the left and </A></LI> for the right and then just combine your search term with a space before the word you scraped? I tried it and it worked for me.

 

Dave

 

EDIT: You need to have LI as the starting attribute for the page scrape and not not A or STRONG which are a couple of the choices I have seen it default to.

 

Hey Dave,

 

I tried just about everything and came up with nothing. Not sure if it's a ubot glitch or a very well crafted webpage.

 

Frank

[Biks 9]

I'm interested in knowing if UBot can scrape the suggested keywords from the Google keyword tool along with Local search volumes per month. Then filter out the ones that are below a certain threshold. (below 3,000 per month)

 

Is this what you guys were talking about?

[Frank 177]

Hey Biks, the next step would be to head over to the external keyword research tool from google and do just that. Pump the keywords in and pull out volume keywords.

 

Frank