mdc101 15 Posted December 3, 2011 Report Share Posted December 3, 2011 Hi Guys I have created a large list of user agentsThe one file scraped by my bot was about 8900.I then ran a test to work on swapping the user agent ui open file("Mobile useragents: ", #MobileUserAgentList)ui open file("Browser user agents:", #BrowserUserAgentList)add list to list(%CurrentUserAgentList, $list from text(#BrowserUserAgentList, $new line), "Delete", "Global")set user agent($random list item(%CurrentUserAgentList))navigate("http://show-ip.net/browserinfo/", "Wait") As soon as we get to set user agent($random list item(%CurrentUserAgentList)) The bot seems to take ages - it does not hang but just does not proceed. I wanted to see how quick the random functions works on large lists. Is there something wrong with the code or cant ubot handle such large lists? I have attached the user agent files if anyone is looking for user agent lists. Thanks for any feedbackmobile-user-agent-list.txtuser-agent-list.txt 2 Quote Link to post Share on other sites
Legend 181 Posted December 3, 2011 Report Share Posted December 3, 2011 Have you tried it with a smaller file to see if the code is sound? Quote Link to post Share on other sites
Bob The Builder 62 Posted December 3, 2011 Report Share Posted December 3, 2011 Hi Guys I have created a large list of user agentsThe one file scraped by my bot was about 8900.I then ran a test to work on swapping the user agent ui open file("Mobile useragents: ", #MobileUserAgentList)ui open file("Browser user agents:", #BrowserUserAgentList)add list to list(%CurrentUserAgentList, $list from text(#BrowserUserAgentList, $new line), "Delete", "Global")set user agent($random list item(%CurrentUserAgentList))navigate("http://show-ip.net/browserinfo/", "Wait") As soon as we get to set user agent($random list item(%CurrentUserAgentList)) The bot seems to take ages - it does not hang but just does not proceed. I wanted to see how quick the random functions works on large lists. Is there something wrong with the code or cant ubot handle such large lists? I have attached the user agent files if anyone is looking for user agent lists. Thanks for any feedbackdo a search I made a snippet that works well for this but much smaller list. Quote Link to post Share on other sites
JohnB 255 Posted December 3, 2011 Report Share Posted December 3, 2011 Try list from file instead of list from text... John Quote Link to post Share on other sites
mdc101 15 Posted December 3, 2011 Author Report Share Posted December 3, 2011 HII have shrunk the file down to about 27 lines and it runs through pretty quickly.What I did notice is it changes the useragent every 5th run. Is this how the random list item works?Is there a way to say grab 25 random urls from a large list text file and then have the used as the useragents and on every 5th run it grabs a new set of say 25 random urls? Ideally one would think that random list item would be every run. Quote Link to post Share on other sites
JohnB 255 Posted December 3, 2011 Report Share Posted December 3, 2011 Ok, the problem is the size of the list...if you watch this, I am using the whole list and the switch useragent is where it gets stuck: http://screencast.com/t/I0Ig3Hsae When I switched it to only use 300 of the list items it works fine: http://screencast.com/t/hmcnXv4EH3 You will notice my setup in the video does not use random. I set up my own random function simply so I could control the size of the list without changing it. The entire list is loaded, but I am only using the first 300 items in the second video. It works just as well with $random list item. (which is the correct way to do it) John clear list(%useragents)add list to list(%useragents, $list from file("{$special folder("Desktop")}\\user-agent-list.txt"), "Delete", "Global")loop(25) { set(#position, $rand(0, 299), "Global") set user agent($list item(%useragents, #position)) navigate("http://whatsmyuseragent.com/", "Wait") wait(4)} Quote Link to post Share on other sites
mdc101 15 Posted December 3, 2011 Author Report Share Posted December 3, 2011 Ah very clever! Thanks John that's a cool way of working with a large list where you need randomness. Thanks guys for the feedback, appreciated Quote Link to post Share on other sites
Alliance 0 Posted February 13, 2012 Report Share Posted February 13, 2012 Hi I am new in this forum, and also in the Ubot world. I am setting my first bot and discovered this ppost to be usefull for me, but unfortunatelly I can't download the useragent list attached. Is there an special precedure to download files in this forum, it says forbitten acces for me, but I am logged in the forum. Thanks for your help. Quote Link to post Share on other sites
toothless 0 Posted March 21, 2012 Report Share Posted March 21, 2012 Hi I am new in this forum, and also in the Ubot world. I am setting my first bot and discovered this ppost to be usefull for me, but unfortunatelly I can't download the useragent list attached. Is there an special precedure to download files in this forum, it says forbitten acces for me, but I am logged in the forum. Thanks for your help.I have the same problem,What can I do also,very nice post Quote Link to post Share on other sites
a2mateit 395 Posted March 22, 2012 Report Share Posted March 22, 2012 I have the same problem,What can I do also,very nice post You need to PM a moderator to get your restriction lifted... Quote Link to post Share on other sites
Website Rich 1 Posted November 7, 2012 Report Share Posted November 7, 2012 Why are all the attachments down on every post I visit? How do we download the ubots and txt files if they are all missing? Or is there a level you need to be before you're able to download any files? Please advise me on what I need to do. Thanks Quote Link to post Share on other sites
Legend 181 Posted November 7, 2012 Report Share Posted November 7, 2012 Why are all the attachments down on every post I visit? How do we download the ubots and txt files if they are all missing? Or is there a level you need to be before you're able to download any files? Please advise me on what I need to do. Thanks Are you registered for this forum? If not just send in a support ticket to axx the goodies... Quote Link to post Share on other sites
Twitter Dominator 125 Posted November 8, 2012 Report Share Posted November 8, 2012 im registered and their not loading for me either, but anyways the code is pasted for a copy paste in a few posts above so you dont need the lists to make it work you can scrape the useragent list from almost everywhere now, the code works in the abvove posthttp://www.ubotstudio.com/forum/index.php?/topic/8641-swapping-user-agents/#entry44072 Quote Link to post Share on other sites
whoami 26 Posted December 19, 2012 Report Share Posted December 19, 2012 This is a good topic, too bad the user agent list he uploaded is not up anymore.Anything we can do? Quote Link to post Share on other sites
Code Docta (Nick C.) 638 Posted January 21, 2013 Report Share Posted January 21, 2013 Anyone have this list to re upload for us? would like to have!! Quote Link to post Share on other sites
AutomationNinja 194 Posted January 31, 2013 Report Share Posted January 31, 2013 user_pref("useragentswitcher.1.appname", ""); user_pref("useragentswitcher.1.appversion", ""); user_pref("useragentswitcher.1.description", ":: BROWSERS - Windows ::"); user_pref("useragentswitcher.1.useragent", ""); user_pref("useragentswitcher.10.description", "Multizilla 1.6 (Win xp)"); user_pref("useragentswitcher.10.useragent", "Mozilla/5.0 (Windows; U; Windows XP) Gecko MultiZilla/1.6.1.0a"); user_pref("useragentswitcher.11.description", "Netscape 7.1 (Win 98)"); user_pref("useragentswitcher.11.useragent", "Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.4) Gecko Netscape/7.1 (ax)"); user_pref("useragentswitcher.12.description", "Netscape 4.8 (Win XP)"); user_pref("useragentswitcher.12.useragent", "Mozilla/4.8 [en] (Windows NT 5.1; U)"); user_pref("useragentswitcher.13.description", "Netscape 3.01 gold (Win 95)"); user_pref("useragentswitcher.13.useragent", "Mozilla/3.01Gold (Win95; I)"); user_pref("useragentswitcher.14.description", "Netscape 2.02 (Win 95)"); user_pref("useragentswitcher.14.useragent", "Mozilla/2.02E (Win95; U)"); user_pref("useragentswitcher.15.description", ":: SPIDERS - search ::"); user_pref("useragentswitcher.15.useragent", ""); user_pref("useragentswitcher.16.description", "Googlebot 2.1 (New version)"); user_pref("useragentswitcher.16.useragent", "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"); user_pref("useragentswitcher.17.description", "Googlebot 2.1 (Older Version)"); user_pref("useragentswitcher.17.useragent", "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"); user_pref("useragentswitcher.18.description", "Msnbot 1.0 (current version)"); user_pref("useragentswitcher.18.useragent", "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"); user_pref("useragentswitcher.19.description", "Msnbot 0.11 (beta version)"); user_pref("useragentswitcher.19.useragent", "msnbot/0.11 (+http://search.msn.com/msnbot.htm)"); user_pref("useragentswitcher.2.appname", "Microsoft Internet Explorer"); user_pref("useragentswitcher.2.appversion", "4.0 (Compatible; MSIE 6.0; Windows NT 5.1)"); user_pref("useragentswitcher.2.description", "MSIE 6 (Win XP)"); user_pref("useragentswitcher.2.useragent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"); user_pref("useragentswitcher.20.description", "Yahoo Slurp"); user_pref("useragentswitcher.20.useragent", "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"); user_pref("useragentswitcher.21.description", "Ask Jeeves/Teoma"); user_pref("useragentswitcher.21.useragent", "Mozilla/2.0 (compatible; Ask Jeeves/Teoma)"); user_pref("useragentswitcher.22.description", ":: BROWSERS - Mac ::"); user_pref("useragentswitcher.22.useragent", ""); user_pref("useragentswitcher.23.description", "Safari 125.8 (Mac OSX)"); user_pref("useragentswitcher.23.useragent", "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/125.2 (KHTML, like Gecko) Safari/125.8"); user_pref("useragentswitcher.24.description", "Safari 85 (Mac OSX)"); user_pref("useragentswitcher.24.useragent", "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/125.2 (KHTML, like Gecko) Safari/85.8"); user_pref("useragentswitcher.25.description", "MSIE 5.15 (Mac OS 9)"); user_pref("useragentswitcher.25.useragent", "Mozilla/4.0 (compatible; MSIE 5.15; Mac_PowerPC)"); user_pref("useragentswitcher.26.description", "Firefox 0.9 (Mac OSX )"); user_pref("useragentswitcher.26.useragent", "Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7a) Gecko/20040614 Firefox/0.9.0+"); user_pref("useragentswitcher.27.description", "Omniweb563 (Mac OSX)"); user_pref("useragentswitcher.27.useragent", "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-US) AppleWebKit/125.4 (KHTML, like Gecko, Safari) OmniWeb/v563.15"); user_pref("useragentswitcher.28.description", ":: BROWSERS - linux/unix/beos ::"); user_pref("useragentswitcher.28.useragent", ""); user_pref("useragentswitcher.29.description", "Epiphany 1.2 (Linux)"); user_pref("useragentswitcher.29.useragent", "ozilla/5.0 (X11; U; Linux; i686; en-US; rv:1.6) Gecko Epiphany/1.2.5"); user_pref("useragentswitcher.3.appname", "Microsoft Internet Explorer"); user_pref("useragentswitcher.3.appversion", "4.0 (compatible; MSIE 5.5; Windows NT 5.0)"); user_pref("useragentswitcher.3.description", "MSIE 5.5 (Win 2000)"); user_pref("useragentswitcher.3.useragent", "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0 )"); user_pref("useragentswitcher.30.description", "Epiphany 1.4 (Ubuntu)"); user_pref("useragentswitcher.30.useragent", "Mozilla/5.0 (X11; U; Linux i586; en-US; rv:1.7.3) Gecko/20040924 Epiphany/1.4.4 (Ubuntu)"); user_pref("useragentswitcher.31.description", "Firefox 0.8 (Linux)"); user_pref("useragentswitcher.31.useragent", "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040614 Firefox/0.8"); user_pref("useragentswitcher.32.description", "Galeon 1.3 (Linux)"); user_pref("useragentswitcher.32.useragent", "Mozilla/5.0 (X11; U; Linux; i686; en-US; rv:1.6) Gecko Galeon/1.3.14"); user_pref("useragentswitcher.33.description", "Konqueror 3 rc4 (Linux)"); user_pref("useragentswitcher.33.useragent", "Konqueror/3.0-rc4; (Konqueror/3.0-rc4; i686 Linux;;datecode)"); user_pref("useragentswitcher.34.description", "Konqueror 3.3 (Gentoo)"); user_pref("useragentswitcher.34.useragent", "Mozilla/5.0 (compatible; Konqueror/3.3; Linux 2.6.8-gentoo-r3; X11;"); user_pref("useragentswitcher.35.description", "Mozilla 1.6 (Debian)"); user_pref("useragentswitcher.35.useragent", "Mozilla/5.0 (X11; U; Linux; i686; en-US; rv:1.6) Gecko Debian/1.6-7"); user_pref("useragentswitcher.36.description", "Opera 7.23 (Linux)"); user_pref("useragentswitcher.36.useragent", "MSIE (MSIE 6.0; X11; Linux; i686) Opera 7.23"); user_pref("useragentswitcher.37.description", "ELinks 0.9.3 (Kanotix)"); user_pref("useragentswitcher.37.useragent", "ELinks/0.9.3 (textmode; Linux 2.6.9-kanotix-8 i686; 127x41)"); user_pref("useragentswitcher.38.description", "Elinks 0.4pre5 (Linux)"); user_pref("useragentswitcher.38.useragent", "ELinks (0.4pre5; Linux 2.6.10-ac7 i686; 80x33)"); user_pref("useragentswitcher.39.description", "Links 2.1 (Linux)"); user_pref("useragentswitcher.39.useragent", "Links (2.1pre15; Linux 2.4.26 i686; 158x61)"); user_pref("useragentswitcher.4.description", "MSIE 5.5 (Win ME)"); user_pref("useragentswitcher.4.useragent", "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"); user_pref("useragentswitcher.40.description", "Links 0.9.1 (Linux)"); user_pref("useragentswitcher.40.useragent", "Links/0.9.1 (Linux 2.4.24; i386;)"); user_pref("useragentswitcher.41.description", "Lynx 2.8.5rel.1 (Linux)"); user_pref("useragentswitcher.41.useragent", "Lynx/2.8.5rel.1 libwww-FM/2.14 SSL-MM/1.4.1 GNUTLS/0.8.12"); user_pref("useragentswitcher.42.description", "w3m 0.5.1 (Linux)"); user_pref("useragentswitcher.42.useragent", "w3m/0.5.1"); user_pref("useragentswitcher.43.description", "Links 2.1 (FreeBSD)"); user_pref("useragentswitcher.43.useragent", "Links (2.1pre15; FreeBSD 5.3-RELEASE i386; 196x84)"); user_pref("useragentswitcher.44.description", "Mozilla 1.7 (FreeBSD)"); user_pref("useragentswitcher.44.useragent", "Mozilla/5.0 (X11; U; FreeBSD; i386; en-US; rv:1.7) Gecko"); user_pref("useragentswitcher.45.description", "Netscape 4.77 (Irix)"); user_pref("useragentswitcher.45.useragent", "Mozilla/4.77 [en] (X11; I; IRIX;64 6.5 IP30)"); user_pref("useragentswitcher.46.description", "Netscape 4.8 (SunOS)"); user_pref("useragentswitcher.46.useragent", "Mozilla/4.8 [en] (X11; U; SunOS; 5.7 sun4u)"); user_pref("useragentswitcher.47.description", "Net Positive 2.1 (BeOS)"); user_pref("useragentswitcher.47.useragent", "Mozilla/3.0 (compatible; NetPositive/2.1.1; BeOS)"); user_pref("useragentswitcher.48.description", ":: SPIDERS - misc ::"); user_pref("useragentswitcher.48.useragent", ""); user_pref("useragentswitcher.49.description", "download demon"); user_pref("useragentswitcher.49.useragent", "Download Demon/3.5.0.11"); user_pref("useragentswitcher.5.appname", ""); user_pref("useragentswitcher.5.appversion", ""); user_pref("useragentswitcher.5.description", "Avant Browser 1.2"); user_pref("useragentswitcher.5.useragent", "Avant Browser/1.2.789rel1 (http://www.avantbrowser.com)"); user_pref("useragentswitcher.50.description", "Email Wolf"); user_pref("useragentswitcher.50.useragent", "EmailWolf 1.00"); user_pref("useragentswitcher.51.description", "grub client"); user_pref("useragentswitcher.51.useragent", "grub-client-1.5.3; (grub-client-1.5.3; Crawl your own stuff with http://grub.org)"); user_pref("useragentswitcher.52.description", "gulperbot"); user_pref("useragentswitcher.52.useragent", "Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)"); user_pref("useragentswitcher.53.description", "ms url control"); user_pref("useragentswitcher.53.useragent", "Microsoft URL Control - 6.00.8862"); user_pref("useragentswitcher.54.description", "omni web"); user_pref("useragentswitcher.54.useragent", "OmniWeb/2.7-beta-3 OWF/1.0"); user_pref("useragentswitcher.55.description", "winHTTP"); user_pref("useragentswitcher.55.useragent", "SearchExpress"); user_pref("useragentswitcher.56.description", ":: v: 1.0.4 :: techpatterns.com ::"); user_pref("useragentswitcher.56.useragent", ""); user_pref("useragentswitcher.56.vendor", "http://techpatterns.com/forums/about304.html"); user_pref("useragentswitcher.6.appname", "Microsoft Internet Explorer"); user_pref("useragentswitcher.6.appversion", "4.0 (compatible; MSIE 6.0; Windows NT 5.0; en)"); user_pref("useragentswitcher.6.description", "Opera 8.0 (Win 2000)"); user_pref("useragentswitcher.6.useragent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; en) Opera 8.0"); user_pref("useragentswitcher.7.description", "Opera 7.51 (Win XP)"); user_pref("useragentswitcher.7.useragent", "Opera/7.51 (Windows NT 5.1; U) [en]"); user_pref("useragentswitcher.8.description", "Opera 7.5 (Win XP)"); user_pref("useragentswitcher.8.useragent", "Opera/7.50 (Windows XP; U)"); user_pref("useragentswitcher.9.description", "Opera 7.5 (Win ME)"); user_pref("useragentswitcher.9.useragent", "Opera/7.50 (Windows ME; U) [en]"); user_pref("useragentswitcher.import.overwrite", false); user_pref("useragentswitcher.menu.hide", false); user_pref("useragentswitcher.reset.onclose", true); user_pref("useragentswitcher.user.agents.count", 56); Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.