Net66 54 Posted March 7, 2011 Report Share Posted March 7, 2011 Google seems to be getting a lot tighter on its detecting of bots with it triggering a capthcha within a few pages even with delays in place. Just wondered if others have noticed this? Its got to the point that I've changed my commercial bots to use bing instead where possible. Andy Quote Link to post Share on other sites
Praney Behl 314 Posted March 7, 2011 Report Share Posted March 7, 2011 You are not wrong Andy, I have noticed the same thing, I get catcha now even when I surf the net and instantly as I do an advanced search. I am also preferring Bing over Google for results. Praney Quote Link to post Share on other sites
UBotBuddy 331 Posted March 7, 2011 Report Share Posted March 7, 2011 I hardly ever use Google. I think they are too busy shooting their own feet to care what customers think. Quote Link to post Share on other sites
JohnB 255 Posted March 7, 2011 Report Share Posted March 7, 2011 Definitely another change for the worse... Quote Link to post Share on other sites
Abs* 12 Posted March 7, 2011 Report Share Posted March 7, 2011 Yup - I copy that - Google is getting extremly tricky to work with I am going to try and code Google in a different way - Will make the search and then scrape the urls to page 1,2,3,4 etc then navigate to each url and scrape - this should save from hitting the next button and making a new query - Not sure if it will work - but hay I could share when its complete - or if anyone else does it then would be great to share thanks abs Quote Link to post Share on other sites
Abs* 12 Posted March 7, 2011 Report Share Posted March 7, 2011 here you go guys try this one out I scraped over 28 pages with no ban and no delay :-) you may need to play with it still as i done it real quick but should give you an idea of how im trying to minimize queries alltogether without hitting the next button as I believe this triggers google. ThanksGoogleScraperABS.ubot 2 Quote Link to post Share on other sites
Net66 54 Posted March 7, 2011 Author Report Share Posted March 7, 2011 Nice one Abs :-) Quote Link to post Share on other sites
meter 145 Posted March 8, 2011 Report Share Posted March 8, 2011 Interesting that it's detecting you guys. Scraping via sockets works fine. Just increment the page count in the URL. If you clear cookies between requests and randomize your useragent you can get a hundreds or so requests in before the ban. That's why you should randomly rotate proxies for each request... Enough proxies, and not a single one will get banned because each will be rotated out before it can reach the ban threshold. -meter Quote Link to post Share on other sites
Frank 177 Posted March 8, 2011 Report Share Posted March 8, 2011 yea, but google is becoming a pain in the behind. I think that they should just create an interface for use bot guys with a simple search window - plain jane - and it doens't effect their monthly search numbers on terms. I've even pay a subscription for such a service and I'm sure that there's others who would too! Easy money maker for Google as well. Frank Quote Link to post Share on other sites
Natureboy 3 Posted April 13, 2011 Report Share Posted April 13, 2011 i used to have similar problems...then i got scrapebox...it scrapes google,yahoo,and bing faster than ubot ever will be able to...get ur list...clean it...then throw it in ubot Quote Link to post Share on other sites
LoWrIdErTJ - BotGuru 904 Posted April 14, 2011 Report Share Posted April 14, 2011 i used to have similar problems...then i got scrapebox...it scrapes google,yahoo,and bing faster than ubot ever will be able to...get ur list...clean it...then throw it in ubot Never say never there bud.Ubot is beginning to change a great deal of things as well as making things much faster, especially with the addition of Sockets will change the speed at which things can be done. Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.