Removing Duplicates items on a list of urls

[southsider 0]

I am scraping a site with a couple different keywords for urls.

 

The problem is my list output has some of the same urls listed multiple times. I need a way to filter or clean the list so the url is just listed once.

 

Anyone have any tips or code to do this, I haven't though a way of doing it yet.

 

Thanks in Advance.

[crazyflx 22]

I am scraping a site with a couple different keywords for urls.

 

The problem is my list output has some of the same urls listed multiple times. I need a way to filter or clean the list so the url is just listed once.

 

Anyone have any tips or code to do this, I haven't though a way of doing it yet.

 

Thanks in Advance.

 

Are they literally identical? (I.E. URL 1=http://somesite.com/ URL 2=http://somesite.com/ )

 

If they are perfectly identical, you can use the "remove duplicates" option on the "add to list" screen.

[southsider 0]

Are they literally identical? (I.E. URL 1=http://somesite.com/ URL 2=http://somesite.com/ )

 

If they are perfectly identical, you can use the "remove duplicates" option on the "add to list" screen.

 

Wow, now do I feel stupid, I have been trying to figure out how to do that for a week, and it is already built into ubot.

 

Thanks for the tip.

[crazyflx 22]

Wow, now do I feel stupid, I have been trying to figure out how to do that for a week, and it is already built into ubot.

 

Thanks for the tip.

 

I've had that exact same feeling about a hundred times while learning to use uBot...I can't even list all the examples there are so many.

 

I've asked so many questions only to find out that the developers knew those questions would be asked and built a solution right into uBot.

 

I've got to say though it sure is a good problem to have, haha.