Good methods for avoiding redundant data scraping?

[ayhoung 0]

Hi guys and experts!

 

I was just hoping to pick your brains here for good methods to speeding up scraping and avoiding redundant information.

 

So let's say a site is continuously updating with new information and has pages and pages of it.

 

Two questions:

1. Can you scrape every individual page as a separate thread to increase speed for scraping?

2. Any good methods of avoiding pages that have already been scraped? What methods do you use?

 

Hoping for interesting insight!

 

Thanks,

Allen

[AutomationNinja 194]

With the pro and dev version you can multi-thread

You could record the urls you have gone to and add them to a list and then compare against that list .....

[ayhoung 0]

"With the pro and dev version you can multi-thread"
 

As far as I understand, multi-threading does the same thing in multiple threads, like creating an account. Can you input variables into different threads such as page numbers for them to scrape different pages?

[AutomationNinja 194]

yes you can

[Code Docta (Nick C.) 638]

You can try this, it basically compares the list and subtracts what has been posted/done and leaves you with what has not neen posted/done.

http://wiki.ubotstudio.com/wiki/$subtract_lists

 

HTH too.