Jump to content



Photo

Best Option - Ubot Browser Or Some Plugin? Ebay And Horrendous Captcha's


  • Please log in to reply
7 replies to this topic

#1 horsemaster

horsemaster

    Newbie

  • Members
  • Pip
  • 5 posts
  • OS:Windows 7
  • Total Memory:8Gb
  • Framework:v3.5
  • License:Developer Edition

Posted 23 April 2020 - 02:51 AM

Been a while off ubot and just came back to it as I need to do some personal automation on my eBay account.

 

I had an older version of ubot and ebay complained about the browsers being out of date.

 

Updated ubot and now the browser works but just get continuous horrendous Captcha's showing making it unusable. Not sure how it's doing this - assume it can detect some sort of browser fingerprint. I'm running it on my normal machine and never get captcha's.

 

I know there are a ton of work arounds with plugins ect...I think there was something called exbrowser? Was/is that still a thing? Guess it would be best do to calls to an external browser?

 

Can anyone point me in the direction of what is the latest best option for dealing with bigger websites that have these captcha systems in place. Or is it a case of just dealing with the captcha's?

 

Don't need anything more than a point in the right direction. What are you guys using daily for web automation in ubot. The built in browser or some plugin?

 

Sorry about the posible noob question



#2 stephenzeiner

stephenzeiner

    Advanced Member

  • Members
  • PipPipPip
  • 32 posts
  • OS:Mac OS
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 07 May 2020 - 11:33 AM

I had an older version of ubot and ebay complained about the browsers being out of date.

 

Updated ubot and now the browser works but just get continuous horrendous Captcha's showing making it unusable. Not sure how it's doing this - assume it can detect some sort of browser fingerprint. I'm running it on my normal machine and never get captcha's.

 

Guess it would be best do to calls to an external browser?

 

Let me be sure I understand: 

 

1) You're able to successfully visit the eBay website from your machine using your day-to-day web browser, whatever browser that happens to be. You can perform many of the routine tasks that you'd typically perform, and at no point are you presented with a captcha roadblock.

 

2) When you tried visiting the eBay website using an older version of UBot Studio, the eBay website provided you with prompts and warnings indicating that the browser through which you were accessing the eBay website (meaning, the UBot Studio built-in browser) was out of date. Presumably at this point, the primary message you were getting from the eBay website was pertaining to the browser being out of date, and at no point did you experience any captcha roadblocks.

 

3) You updated to the latest version of UBot Studio v6+. You're able to successfully visit the eBay website using the built-in UBot Studio browser, however, the eBay website is now presenting you with captcha roadblocks while using the UBot Studio built-in browser.

 

Something that's unclear from your post is whether or not you're using the UBot Studio v6+ built-in browser as a human, meaning, that you're manually visiting the eBay website by typing the URL into the UBot Studio v6+ browser navigation window and then attempting to use the eBay website -- only to suddenly be presented with captcha roadblocks. Or alternatively, are you running a high velocity multi-threaded UBot Studio v6+ program that's accessing the eBay website in a rapid-fire manner and only then are you noticing captcha roadblocks.

 

It could be useful to visit the eBay website through the UBot Studio v6+ internal browser and attempt to just use the site as you ordinarily would, typing and clicking as you go along. See if that triggers any captcha roadblocks. If it does, then it suggests there's something being communicated to the eBay website server by the UBot Studio v6+ built-in browser that is triggering the captcha roadblock.

 

And also, yes, as you suggested, try using the UBot Studio v6+ external browser commands to determine whether or not that makes a difference.

 

I'm genuinely interested in reading what your testing reveals.



#3 LoWrIdErTJ - BotGuru

LoWrIdErTJ - BotGuru

    Botguru.net

  • Moderators
  • 4844 posts
  • LocationMichigan
  • OS:Windows 10
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 08 May 2020 - 06:48 PM

They show captcha because unlike your daily browser you have no cookies, cache, history in your ubot browser.

#4 stephenzeiner

stephenzeiner

    Advanced Member

  • Members
  • PipPipPip
  • 32 posts
  • OS:Mac OS
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 24 May 2020 - 02:09 PM

They show captcha because unlike your daily browser you have no cookies, cache, history in your ubot browser.

 

That's an interesting and through provoking insight.

 

Your insight led me to wonder about two things that I don't know much about, but that I am genuinely interested in learning more about:

 

1) How does eBay (or any website, for that matter):

 

a ) determine the content and number of cookies that are stored on a user's computer system,

b ) determine the content and size of various cache files stored on a user's computer system, and

c ) read the web browsing history contained in whatever web browser a user happens to be using.

 

2) Would a user who routinely clears their browser cookies/cache/history be subjected to Captcha roadblocks when visiting eBay?

 

I'm definitely going to dig into this because it's unclear to me that most people would even have the faintest idea that their cookies, cache, and browsing history is readily accessible to eBay for analysis by their system to help determine if Captcha roadblocks should be presented. I certainly wasn't aware of this, if true.

 

Prior to posting this reply I took a quick look at eBay's published 'User Privacy Notice' but didn't immediately spot anything conclusive.

 

I'll do some testing in the days ahead and see what I discover.

 

This truly does intrigue me.

 

[Edited to change the way the letter "b" followed by a parenthesis ")" symbol was displayed so that the forum software wouldn't turn it into a sunglasses wearing emoji with a crooked smile.]


Edited by stephenzeiner, 24 May 2020 - 02:11 PM.


#5 HelloInsomnia

HelloInsomnia

    Advanced Member

  • Moderators
  • 3307 posts
  • OS:Windows 10
  • Total Memory:More Than 9Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 24 May 2020 - 02:38 PM

Ebay ports scans your computer to determine if you're using remote access programs, e.g. remote desktop. Something to keep in mind here (may or may not be related to your issue but thought I'd mention it).



#6 stephenzeiner

stephenzeiner

    Advanced Member

  • Members
  • PipPipPip
  • 32 posts
  • OS:Mac OS
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 25 May 2020 - 09:56 PM

Ebay ports scans your computer to determine if you're using remote access programs, e.g. remote desktop. Something to keep in mind here (may or may not be related to your issue but thought I'd mention it).

 

I just stumbled onto a story on Slashdot that mentions the surprising phenomenon you highlighted:

https://it.slashdot....access-programs

 

I've yet to conduct the testing mentioned in my previous message, but at this point, I wouldn't be too radically surprised if I discovered eBay also activates any webcam on a user's computer system and records user facial expressions and movements while the eBay website is being used. :-)

 

(Okay, just kidding about the webcam part, but still... if eBay is actually gaining access to all the cookies on a user's system, browser cache files, web browsing history, *and* port scanning... I mean, it paints a somewhat Orwellian picture that could plausibly include webcam/microphone tapping as well -- although, realistically, I can't imagine that would be happening.)



#7 stephenzeiner

stephenzeiner

    Advanced Member

  • Members
  • PipPipPip
  • 32 posts
  • OS:Mac OS
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 31 May 2020 - 02:31 PM

I'm back with some test results, as well as with the result of some research I conducted.
 
- - - - - - 
TL;DR: I neither could replicate having eBay present me with a complaint that my browser was out of date (despite using Chrome 49 on UBot Studio v5.9.55) nor could I replicate eBay crippling my experience on their site with Captcha's despite browsing around their site for over 10+ minutes. I genuinely believe the OP's experience on eBay *is* being crippled by Captcha's, I just couldn't get eBay to cripple my experience. I cleared cookies before starting to use the eBay site. Here's a sped up version of my experience captured via Screencast-O-Matic: https://www.youtube....h?v=GdjgQZHhQWs
- - - - - -  
 
Some questions that I wondered about in a previous post that I wanted to try and get some answers to that might be helpful to others.
 
1) How does eBay (or any website, for that matter):
 
a ) determine the content and number of cookies that are stored on a user's computer system,
 
It does not (not shouting, just bolding for emphasis in a sea of text) appear as though eBay can determine the content and number of cookies that are stored on a user's computer system except for cookies that may have been set by eBay itself or by third-parties which collaborate with eBay. 
(Source: https://security.sta...r-pages-cookies

Here's more information regarding the exception for situations where "syndicated sites" are involved:
 

First yes. Simply put, if you allow a site to save your cookies, they will often share them with other syndicated sites, so that when you go to other syndicated sites, you will be prioritized to show what you've searched for.
If you want to avoid this, there are three ways:
1.Use browser focused on privacy(eg: Private Browser). Using Private Browser will keep you from being tracked, and once you exit, all of your records and passwords will be erased by default, which is the most convenient method.
2.Turn on privacy mode of the normal browser(eg:Chrome,UC,Firefox). Main browsers generally have privacy mode, and you can usually find it directly from the menu or the bottom bar and turn on it. But some also reflect that these modes are not completely private.
3.Keep Clear Browsing history. It means that each time you exit the browser, you need to go to setting>clear data. It's relatively cumbersome.

 
(Source: https://superuser.co...rowsing-history)

b ) determine the content and size of various cache files stored on a user's computer system, and

Maybe I wasn't sure how to research this the right way, but I couldn't find any information about how a website (be it eBay or any other site) could somehow "scan" a user's system to determine the content and size of various cache files. It's entirely plausible this happens every time someone visits eBay, I just couldn't find any information on how this would be done.

c ) read the web browsing history contained in whatever web browser a user happens to be using.

I found a very similar question to this on StackExchange, which reads in part:
 

When I visit a website it sometimes puts cookies on my browser. Usually cookies keep track of browsing info from that particular site. If I don't delete those cookies and browse other websites, will the cookies be able to know which site I am browsing? Also, can any website read browsing history which is saved by browsers? Can any website know about my browsing history by help of cookies, browsing histories saved by browsers, or by any scripting language e.g. javascript?

 
The relevant reply reads in part:
 

The short answer is yes, though it is not as easy as you might think.
The browser stores cookies independently for each domain. That means that www.foo.com cannot access the cookies made by www.bar.com and vice versa.
The vulnerability (or loophole) is in included pages. Most advertisements come from a different domain than the page itself, so they create their own sets of cookies. When another site includes an ad from the same ad provider, they can read their own cookies created earlier and they know you visited that page earlier. This way they can only track you on sites that host their ads. This is the strategy Google uses to serve relevant ads.
Also Facebook and other social networks can do this because of their ubiquitous like, tweeet, pin etc. buttons, which are also included content. This is not avoidable without disabling cookies altogether or using private browsing, but that could be a major burden (Cookies do make the Internet convenient). I personally choose not to be paranoid of these things.

 
(Source: https://superuser.co...rowsing-history)

So it appears as though *some* degree of surveillance is possible -- not through the ability to scan every cookie and one's complete browsing history (or at least that's not my understanding based on the reply), but by being able to piece together information based on some of the technology described in the aforementioned StackExchange reply. 

2) Would a user who routinely clears their browser cookies/cache/history be subjected to Captcha roadblocks when visiting eBay?

Potentially, yes.

This is from the 'Google reCAPTCHA: Prevention of automated access' section of the eBay website:
 

1.5 Google reCAPTCHA: Prevention of automated access
 
We make use of Google reCAPTCHA ("reCAPTCHA") within the provision of our Services. reCAPTCHA is used to check and prevent interactions through automated access, for example through so-called bots (computer programs that perform tasks automatically and independently). reCAPTCHA is used in particular to check whether data input in connection with our Services (e.g. in a contact form) is carried out by a human or by an automated program. For this purpose, reCAPTCHA analyzes the behavior of the user on the basis of various characteristics. This analysis automatically begins as soon as the user connects with the online service, e.g. visits the website. For the purposes of this analysis, reCAPTCHA evaluates various pieces of information (e.g. IP address, duration of the user's visit to the online service or the user's mouse movements). The data collected during the analysis is forwarded to Google. The analyses by reCAPTCHA are carried out entirely in the background.

 
(Source: https://www.ebay.com...-notice?id=4267)
 
It appears as though the triggering of reCaptcha, which is what eBay appears to use, is now heavily dependent on a set of subjective in-house criteria that can be established by websites to determine what constitutes so-called "suspicious" behavior -- based on a "score" provided to websites by Google.
 
This is a great description of the evolution of the Google reCAPTCHA system as outlined on the Google Webmaster Central Blog:
 

Over the last decade, reCAPTCHA has continuously evolved its technology. In reCAPTCHA v1, every user was asked to pass a challenge by reading distorted text and typing into a box. To improve both user experience and security, we introduced reCAPTCHA v2 and began to use many other signals to determine whether a request came from a human or bot. This enabled reCAPTCHA challenges to move from a dominant to a secondary role in detecting abuse, letting about half of users pass with a single click. Now with reCAPTCHA v3, we are fundamentally changing how sites can test for human vs. bot activities by returning a score to tell you how suspicious an interaction is and eliminating the need to interrupt users with challenges at all. reCAPTCHA v3 runs adaptive risk analysis in the background to alert you of suspicious traffic while letting your human users enjoy a frictionless experience on your site.

 
(Source: https://webmasters.g...new-way-to.html)
 
# # #
 
I'm actually really happy that I dug into this because the process of researching some of these questions helped educate me about several things that I really hadn't quite understood. In some ways, I'm still not entirely sure that I fully understand some of these things.



#8 luis carlos

luis carlos

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 492 posts
  • OS:Windows 8
  • Total Memory:< 1Gb
  • Framework:v3.5
  • License:Developer Edition

Posted 01 June 2020 - 03:29 PM

I'm back with some test results, as well as with the result of some research I conducted.
 
- - - - - - 
TL;DR: I neither could replicate having eBay present me with a complaint that my browser was out of date (despite using Chrome 49 on UBot Studio v5.9.55) nor could I replicate eBay crippling my experience on their site with Captcha's despite browsing around their site for over 10+ minutes. I genuinely believe the OP's experience on eBay *is* being crippled by Captcha's, I just couldn't get eBay to cripple my experience. I cleared cookies before starting to use the eBay site. Here's a sped up version of my experience captured via Screencast-O-Matic: https://www.youtube....h?v=GdjgQZHhQWs
- - - - - -  
 
Some questions that I wondered about in a previous post that I wanted to try and get some answers to that might be helpful to others.
 
1) How does eBay (or any website, for that matter):
 
a ) determine the content and number of cookies that are stored on a user's computer system,
 
It does not (not shouting, just bolding for emphasis in a sea of text) appear as though eBay can determine the content and number of cookies that are stored on a user's computer system except for cookies that may have been set by eBay itself or by third-parties which collaborate with eBay. 
(Source: https://security.sta...r-pages-cookies

Here's more information regarding the exception for situations where "syndicated sites" are involved:
 

 
(Source: https://superuser.co...rowsing-history)

b ) determine the content and size of various cache files stored on a user's computer system, and

Maybe I wasn't sure how to research this the right way, but I couldn't find any information about how a website (be it eBay or any other site) could somehow "scan" a user's system to determine the content and size of various cache files. It's entirely plausible this happens every time someone visits eBay, I just couldn't find any information on how this would be done.

c ) read the web browsing history contained in whatever web browser a user happens to be using.

I found a very similar question to this on StackExchange, which reads in part:
 

 
The relevant reply reads in part:
 

 
(Source: https://superuser.co...rowsing-history)

So it appears as though *some* degree of surveillance is possible -- not through the ability to scan every cookie and one's complete browsing history (or at least that's not my understanding based on the reply), but by being able to piece together information based on some of the technology described in the aforementioned StackExchange reply. 

2) Would a user who routinely clears their browser cookies/cache/history be subjected to Captcha roadblocks when visiting eBay?

Potentially, yes.

This is from the 'Google reCAPTCHA: Prevention of automated access' section of the eBay website:
 

 
(Source: https://www.ebay.com...-notice?id=4267)
 
It appears as though the triggering of reCaptcha, which is what eBay appears to use, is now heavily dependent on a set of subjective in-house criteria that can be established by websites to determine what constitutes so-called "suspicious" behavior -- based on a "score" provided to websites by Google.
 
This is a great description of the evolution of the Google reCAPTCHA system as outlined on the Google Webmaster Central Blog:
 

 
(Source: https://webmasters.g...new-way-to.html)
 
# # #
 
I'm actually really happy that I dug into this because the process of researching some of these questions helped educate me about several things that I really hadn't quite understood. In some ways, I'm still not entirely sure that I fully understand some of these things.

Thank you for this information, really valuable.

 

 

 

Luis Carlos






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users