Scraping 3 Copies

[Legend 181]

I've run into this before and haven't found a solution yet. Quite often when I scrape a variable it returns 3 copies of the same variable, or duplicates it 3 times. Anyone else running into this? Is there a quick cure?

 

Thanks,

Duane

[UBotBuddy 331]

Give me a site to look at.

[Tomas 0]

Give me a site to look at.

Same problem here.

 

clear list(%URL_list)
add item to list(%URL_list, $scrape attribute(<tagname="a">, "href"), "Delete", "Global")
save to file("{$special folder("Application")}/urls.txt", %URL_list)

url scraped http://www.lrytas.lt/kiti/rss.htm

 

But only some links 3 times included. But function Delete Duplicates was selected.

 

urls.txt

[UBotBuddy 331]

Tomas,

 

The reason you are getting so many is that your scrape is wrong.

 

Let me see if I can refine it.

[UBotBuddy 331]

Here ya go.

 

clear list(%URL_list)

add list to list(%URL_list, $scrape attribute(*">, "href"), "Delete", "Global")

save to file("{$special folder("Application")}/urls.txt", %URL_list)

[UBotBuddy 331]

For the Delete to work you needed to use the "add list to list ". When you were scraping you were adding One item to the list. So as one item it was unique.

[Tomas 0]

For the Delete to work you needed to use the "add list to list ". When you were scraping you were adding One item to the list. So as one item it was unique.

 

Thank you for explanation

[JohnB 255]

Buddy's right...to try and clear it up a little more, if you use add item to list, it may scrape several lines (as viewed in the debugger) that look like a list, but they are not. No matter how many lines are scraped, they are treated as a single item. Sometimes you will run into an issue where it appears more than once in your list because it appears more than once in the document text. One solution is to scrape the multiple items to a list and then set the variable to the first list item (when element offset is not an option).

 

 

John

[Legend 181]

Give me a site to look at.

 

If you have access to a WHM Panel, that is the latest place I encountered it. Using the command:

 

set(#whmloads, $scrape attribute(<innertext=w"Load Averages: *">, "innertext"), "Global")

 

Returns 3 instances of the same results.

 

It happens quite often so I will let you know the next time it happens in a more public place so you can check it out. I was just wondering if anyone else was experiencing it. I just use regex to get rid of the extra stuff <technical term> but it doesn't seem like it should be happening.

 

Cheers,

Duane