southsider 0 Posted March 4, 2010 Report Share Posted March 4, 2010 I am scraping a site with a couple different keywords for urls. The problem is my list output has some of the same urls listed multiple times. I need a way to filter or clean the list so the url is just listed once. Anyone have any tips or code to do this, I haven't though a way of doing it yet. Thanks in Advance. Quote Link to post Share on other sites
crazyflx 22 Posted March 4, 2010 Report Share Posted March 4, 2010 I am scraping a site with a couple different keywords for urls. The problem is my list output has some of the same urls listed multiple times. I need a way to filter or clean the list so the url is just listed once. Anyone have any tips or code to do this, I haven't though a way of doing it yet. Thanks in Advance. Are they literally identical? (I.E. URL 1=http://somesite.com/ URL 2=http://somesite.com/ ) If they are perfectly identical, you can use the "remove duplicates" option on the "add to list" screen. Quote Link to post Share on other sites
southsider 0 Posted March 4, 2010 Author Report Share Posted March 4, 2010 Are they literally identical? (I.E. URL 1=http://somesite.com/ URL 2=http://somesite.com/ ) If they are perfectly identical, you can use the "remove duplicates" option on the "add to list" screen. Wow, now do I feel stupid, I have been trying to figure out how to do that for a week, and it is already built into ubot. Thanks for the tip. Quote Link to post Share on other sites
crazyflx 22 Posted March 4, 2010 Report Share Posted March 4, 2010 Wow, now do I feel stupid, I have been trying to figure out how to do that for a week, and it is already built into ubot. Thanks for the tip. I've had that exact same feeling about a hundred times while learning to use uBot...I can't even list all the examples there are so many. I've asked so many questions only to find out that the developers knew those questions would be asked and built a solution right into uBot. I've got to say though it sure is a good problem to have, haha. Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.