tooltrainer 12 Posted March 23, 2010 Report Share Posted March 23, 2010 I don't know what changed, but all of a sudden Youtube is showing a different layout of their site to UBot than to IE or FF. This is of course breaking the bot that I've spent so much time perfecting. Until this, UBot always saw the same version of Youtube that I do when I visit with a browser. What the heck has happened that would cause UBot to get a different version now? Jonathan Quote Link to post Share on other sites
tooltrainer 12 Posted March 23, 2010 Author Report Share Posted March 23, 2010 OK to make matters stranger, this issue only happen when UBot is running inside a Virtual Machine in VMWare. When I run UBot on my physical machine, the YouTube layout is normal as expected. One difference - the Physical machine is on XP Pro and IE has not been updated, the VM is running Win 7 with the latest IE. Thoughts? Jonathan Quote Link to post Share on other sites
tooltrainer 12 Posted March 24, 2010 Author Report Share Posted March 24, 2010 Well week now for a fact that UBot leaves a footprint, doesn't it identify itself as UBot in the user agent? I remember seeing a thread on that somewhere. But in this case, I don't believe that's what's happening at all. So far I've tested these cases: 1. UBot in physical machine, IE6, normal Youtube layout as expected2. UBot in virtual machine, IE8, strange layout of Youtube that I've never seen3. IE6 manual browser physical machine, normal Youtube layout as expected4. IE8 manual browser virtual machine, normal Youtube layout as expected5. FF6 manual browser virtual machine, normal Youtube layout as expected So it's specifically and only UBot with the virtual machine that is having this problem. I'm sure it's not footprint related. Jonathan Quote Link to post Share on other sites
alcr 135 Posted March 24, 2010 Report Share Posted March 24, 2010 The ability to change the useragent is on our todo. But I'm pretty sure that it doesn't actually leave a UBot footprint. The only thing that can come to my mind is some kind of problem with the javascript hooks. I'll tell Seth to take a peek at it - see if he can find anything obvious. And for tooltrainer: Try updating IE from IE6 Quote Link to post Share on other sites
tooltrainer 12 Posted March 24, 2010 Author Report Share Posted March 24, 2010 fun4uoc - my problem is with Youtube not facebook. And it's not an issue of being banned or blocked or anything. Youtube is showing up in a different layout that I've never seen before. With the description directly underneath the video instead of on the right. I almost think this is some sort of precursor to a wider site change on Youtube's part. Earlier this evening I was working in Ubot on my physical machine rather than the virtual one, and one time (and one time only) the same strange layout appeared. But every other time it was normal and as expected. alcr - I can certainly upgrade my physical machine from IE6 but I'm sure that's not the problem, as it's been working fine on the physical machine in IE6 (except for the one time mentioned above) and has never worked properly once, with IE8 under the VM. Since it normally works fine with IE6 anyway, upgrading to a newer IE on the physical machine won't tell me anything useful since it would be expected to continue to work properly. What I need is a consistent case where I get this new/altered layout, and then another case that causes it to change back to the normal layout. So far I'm striking out though. See the attached screenshot for an example of both layouts on my VM... UBot on the left with video description under the video, IE8 on the right with the standard description on the right side. Jonathan Quote Link to post Share on other sites
alcr 135 Posted March 24, 2010 Report Share Posted March 24, 2010 Hehe well we actually don't even support IE6 But that screenshot is interesting. Aren't youtube updating their interface? Just a guess but it could be that some useragents & IP's = new interface, other useragents & IP's = old interface? Quote Link to post Share on other sites
tooltrainer 12 Posted March 24, 2010 Author Report Share Posted March 24, 2010 Yeah that's my current thinking... that it's an interface change that is only partially rolled out. But there are a couple of things that make me less certain of this: 1) On the VM UBot running Win7 with IE8, the "new interface" is the ONLY one I see, every time, always. I never see it when manually browsing however.2) On my physical machine running XP Pro with IE6, I have seen the "new interface" on only a few occasions (maybe 3) and when it happens I can pause the bot, hit the back button in UBot's browser to go back to the "new interface" video that just showed up, and it will NOT be the "new interface" anymore! Since the UA & IP should be identical in all my cases, I don't think it's UA or IP based. I really can't figure this out... very maddening. Jonathan Quote Link to post Share on other sites
turbolapp 5 Posted March 24, 2010 Report Share Posted March 24, 2010 Well week now for a fact that UBot leaves a footprint, doesn't it identify itself as UBot in the user agent? I remember seeing a thread on that somewhere. But in this case, I don't believe that's what's happening at all. So far I've tested these cases: 1. UBot in physical machine, IE6, normal Youtube layout as expected2. UBot in virtual machine, IE8, strange layout of Youtube that I've never seen3. IE6 manual browser physical machine, normal Youtube layout as expected4. IE8 manual browser virtual machine, normal Youtube layout as expected5. FF6 manual browser virtual machine, normal Youtube layout as expected So it's specifically and only UBot with the virtual machine that is having this problem. I'm sure it's not footprint related. Jonathan How bout ubot in the physical machine higher than IE6 and/or Ubot on VM less than IE 8? I know your frustration, though, too many variables to try to control for and not enough info. Quote Link to post Share on other sites
tooltrainer 12 Posted March 24, 2010 Author Report Share Posted March 24, 2010 Yeah, too many variables. The problem is that the expected behavior on physical machine higher than IE6 would be that everything works fine, which is (mostly) how it is now with IE6. So if I were to upgrade to IE8 and everything still worked, then I've learned nothing. If I were to upgrade to IE8 and it suddenly showed the weird Youtube layout, then I've just completely confused the issue even further and still have no idea what to do about it. So either way the data point wouldn't be helpful. I'm trying to just account for this new layout showing up occasionally but can't seem to get Youbot to properly detect the alternate format. Need to start another thread about that in fact. Jonathan Quote Link to post Share on other sites
Enigma 78 Posted May 18, 2010 Report Share Posted May 18, 2010 The ability to change the useragent is on our todo. But I'm pretty sure that it doesn't actually leave a UBot footprint. The only thing that can come to my mind is some kind of problem with the javascript hooks. I'll tell Seth to take a peek at it - see if he can find anything obvious. And for tooltrainer: Try updating IE from IE6 Any update on this? Would really like to have a useragent faker / randomizer in it. Quote Link to post Share on other sites
alcr 135 Posted May 18, 2010 Report Share Posted May 18, 2010 I will check with Seth Quote Link to post Share on other sites
Net66 54 Posted May 18, 2010 Report Share Posted May 18, 2010 I will check with Seth Cool - the useragent thing will make a big difference. I was chatting to a good friend at the weekend who does the security to stop people scraping data from a big marketing site. He says it is a losing battle and things like capatchas are just making the web more un-user friendly because bots soon get round them (as well we all know) and they just get in the way of normal users. His companies approach is to simply look for repeated hits in a short space of time from an IP or range of IPs, but they have to take into account user agent because they have had universities legitimately request lots of data from a group of different machines but they all share the same IP. They don't want to accidentally ban these. He told me that if I could swap the user agent about and used proxies I'd be able to scrape indefinate amounts of data from their site because as blocks are added it is by IP AND Useragent and they can't really change that much without screwing the use for real users. So roll on the ability to change that about a bit :-) Andy Quote Link to post Share on other sites
Enigma 78 Posted May 18, 2010 Report Share Posted May 18, 2010 I will check with Seth Thanks. A feature to fake the referrer too would be too much asked? Quote Link to post Share on other sites
gproxy 1 Posted July 16, 2010 Report Share Posted July 16, 2010 +1 for multiple useragents. There's a popular FB bot that is apparently getting flagged from it's users using the same useragent spoof. Quote Link to post Share on other sites
Enigma 78 Posted July 21, 2010 Report Share Posted July 21, 2010 Any update on this??? Hope we can do this by simply reading the UserAgents (UA) from a text file so we can all decide which UA we wanna use and if we use $nextlistitem, $randomlistitem, etc. Or even use a single UA depends on what we use. Quote Link to post Share on other sites
Enigma 78 Posted July 23, 2010 Report Share Posted July 23, 2010 I will check with Seth Bump. Any update? Really need this.I hope a solution described here can be achieved http://ubotstudio.com/forum/index.php?/topic/3134-what-is-ubot-identifying-itself-as/page__view__findpost__p__17861 Quote Link to post Share on other sites
Seth Turin 223 Posted July 24, 2010 Report Share Posted July 24, 2010 I've actually tried messing with the user agent before, with little success. Keep in mind that the UBot browser is just an instance of internet explorer, which means that doing this kind of stuff actually means hacking internet explorer. In a lot of cases I've had success getting at IE's innards, but this was not one of those cases. I'll take another look at it though, and see if there is anything else I can do with it. Quote Link to post Share on other sites
Enigma 78 Posted July 24, 2010 Report Share Posted July 24, 2010 I've actually tried messing with the user agent before, with little success. Keep in mind that the UBot browser is just an instance of internet explorer, which means that doing this kind of stuff actually means hacking internet explorer. In a lot of cases I've had success getting at IE's innards, but this was not one of those cases. I'll take another look at it though, and see if there is anything else I can do with it. Hmm... bad news Quote Link to post Share on other sites
iglow 8 Posted July 25, 2010 Report Share Posted July 25, 2010 I have same proble mwith tumblr.com i cant even load site with ubot but with ie i can. Quote Link to post Share on other sites
Praney Behl 314 Posted July 30, 2010 Report Share Posted July 30, 2010 Hey guys I second mutiple useragents.. @SETH I have been doing some research about this, I found this at the imacros wiki: http://wiki.imacros.net/-useragent Is it possible with Ubot?? Secondly, Is there a way to restart the browser within the code to implement new browser setting?? Thanks, Praney Quote Link to post Share on other sites
meter 145 Posted July 31, 2010 Report Share Posted July 31, 2010 Just use SHELL command to change UserAgent...http://www.walkernews.net/2007/07/05/how-to-change-user-agent-string/ Quote Link to post Share on other sites
Praney Behl 314 Posted July 31, 2010 Report Share Posted July 31, 2010 Just use SHELL command to change UserAgent...http://www.walkernews.net/2007/07/05/how-to-change-user-agent-string/ Hey thanks meter, I have already been to this link and also done some extended research about the same but came to a conclusion that you need to restart the browser for the UserAgent to be effective. Any idea on how that can be done?? Cheers!! Praney Quote Link to post Share on other sites
meter 145 Posted August 1, 2010 Report Share Posted August 1, 2010 In that case, Only thing I can think of is have a bat to change the Useragent and then start the Ubot. Not a neat way to do it I'll dig into it further if I get the time. Quote Link to post Share on other sites
Praney Behl 314 Posted August 1, 2010 Report Share Posted August 1, 2010 No worries meter, But thanks for your time.. Quote Link to post Share on other sites
iglow 8 Posted August 5, 2010 Report Share Posted August 5, 2010 I dont want ot use shell i want to have option! Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.