Scraping Dropdown Content

[DjProg 3]

Hello guys,

 

I don't succeed to do what I think must be really simple.

 

On some websites, I found dropdowns with an HTML code like this :

 

<center><select name="id_cat" class="list_cats">
<option value=0>Choose a Category</option>
<option value=0>--------------------------</option>
<option label="Category 1 " value=11>Category 1</option>
<option label="Category 2 " value=5>Category 2</option>
</select>
</center>

 

I "just" want to scrape :

"Category 1 " value=11>Category 1

"Category 2 " value=5>Category 2

 

I tried this :

 

CHOOSE BY ATTRIBUTE

-----InnerHtml

-----<OPTION label=*</OPTION>

-----Wildcards

 

Which is supposed to get the multiple categories as far as i know

 

ADD TO LIST

-----%scraped-categories

-----$scrape chosen attribute

----------innerHTML

 

Which is supposed to put all the scraped categories in a list

 

And it just don't work at all...

 

Could you please help me, i'm kind of desperate and it MUST be simple i'm sure.

 

Cheers,

[Pete 121]

I would think you need more wild cards

<option label="Category * " value=*>*</option>

[UBotBuddy 331]

First. I have done what you are trying to do. But it was a lot of coding on my side that made it happen AND my solution will not work on your site.

 

Here is where you are going to get tripped up.

 

the "

 

Now here is a potential solution

 

 

Choose by attribute the innerhtml let the following be your Search String also select Wildcards

*

 

Then do an Add to List and for your content use the $scrape chose attribute using innerhtml

 

That will get you each of these into your list

 

Category 1

 

Now setup a loop to go through each item in the list and what you want to do is perform a $replace

 

search for "" and replace with Null/nothing

 

perform another $replace

 

search for "

" and replace hat with a Null/nothing

 

You will have to have enough "Category 1", "Category 2", "Category 3" until "Category x" is met.

 

Remember, I am doing this from memory. Not knowing your website to see the actual dropdown limits me to some degree.

 

But this is the essence in how I achieved my scrape.

[DjProg 3]

Hi thanks for your help !

 

I tried but it still doesn't want to collect the categories (i'm just speaking of unformated categories, formating is another story...)

 

You can have a look at the bot attached, in case it inspires you

category-scrapping.ubot

 

I also tried to do a pagescrape but it doesn't work either... (I thought it would work with the following : pagescrape LEFT : <option label= RIGHT : </option> but it doesn't scrape anything...)

 

Cheers,

[UBotBuddy 331]

Hmmmm

 

The only way that I can see to do it is to write some Javascript to extract the text between ">" and ""

 

It can be done just not by me. It would take me a few months. LOL

 

I was able to scrape the contents of the dropdown and your code can too. Just change your Search String in your "choose by attribute" node to *>*

 

When you examine the output file you will see all of the options. Now, had the website coder written the

 

For instance, search for "

" and replace it with "".

 

Sorry that my answer does not get you closer to a solution.

[IRobot 43]

$page scrape does work.

 

See: category-scrapping.ubot

[DjProg 3]

I was able to scrape the contents of the dropdown and your code can too. Just change your Search String in your "choose by attribute" node to *>*</OPTION>

 

It's weird, I thought they way i had written my choose by attribute was correct. I'll try yours.

 

Thanks !

[DjProg 3]

$page scrape does work.

 

See: category-scrapping.ubot

 

Thank you VERY VERY much.

 

I don't know why but the following didn't work :

 

http://screencast.com/t/MWNlOTVhYzkt

[IRobot 43]

Thank you VERY VERY much.

No problem.

 

I don't know why but the following didn't work :

http://screencast.com/t/MWNlOTVhYzkt

It didn't work because the left and right 'option' $page scrape items are lower case, whereas in the working bot I attached, they are upper case.

[UBotBuddy 331]

Interesting! It did not work for me. Our code is identical. Mine does not work and your does.

 

Hmmmmm

[UBotBuddy 331]

Found it. I had been rehashing an open test bot. Had I just Refreshed it the code would have worked.