Cyrax 2 Posted April 30, 2017 Report Share Posted April 30, 2017 i started using ubot again after a long long break. so yesterday i created a bot to scrape from a major real estate directory. i purchased a list of us cities and areas for $49 , the list contains about 45,000 areas my bot is searching the site for realtors based on the area and state code the bot has been running now for 20hrs non stop my file size is currenlty 75mb full of urls i have only another 42,000 areas/keywords searches to make and iam also very shocked because ive been using no proxies, just my normal isp ip ! this may sound a bit odd, but genuinely i feel a bit bad , sad and guilty. Quote Link to post Share on other sites
jason 101 Posted April 30, 2017 Report Share Posted April 30, 2017 Data wants to be free! Quote Link to post Share on other sites
HelloInsomnia 1103 Posted April 30, 2017 Report Share Posted April 30, 2017 this may sound a bit odd, but genuinely i feel a bit bad , sad and guilty. Just try to be respectful of the website you scrape, don't try to pound it with tons of connections or query it several times per second. And if they have an API use that. Otherwise, don't feel bad look at search engines like Google and services like Archive - they both would crawl the whole site and people praise these services.. 1 Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.