Swapping user agents

mdc101 · December 3, 2011

Hi Guys

I have created a large list of user agents

The one file scraped by my bot was about 8900.

I then ran a test to work on swapping the user agent

ui open file("Mobile useragents: ", #MobileUserAgentList)

ui open file("Browser user agents:", #BrowserUserAgentList)

add list to list(%CurrentUserAgentList, $list from text(#BrowserUserAgentList, $new line), "Delete", "Global")

set user agent($random list item(%CurrentUserAgentList))

navigate("http://show-ip.net/browserinfo/", "Wait")

As soon as we get to

set user agent($random list item(%CurrentUserAgentList))

The bot seems to take ages - it does not hang but just does not proceed.

I wanted to see how quick the random functions works on large lists.

Is there something wrong with the code or cant ubot handle such large lists?

I have attached the user agent files if anyone is looking for user agent lists.

Thanks for any feedback

mobile-user-agent-list.txt

user-agent-list.txt

Legend · December 3, 2011

Have you tried it with a smaller file to see if the code is sound?

Bob The Builder · December 3, 2011

Hi Guys

I have created a large list of user agents
The one file scraped by my bot was about 8900.
I then ran a test to work on swapping the user agent

ui open file("Mobile useragents: ", #MobileUserAgentList)
ui open file("Browser user agents:", #BrowserUserAgentList)
add list to list(%CurrentUserAgentList, $list from text(#BrowserUserAgentList, $new line), "Delete", "Global")
set user agent($random list item(%CurrentUserAgentList))
navigate("http://show-ip.net/browserinfo/", "Wait")

As soon as we get to
set user agent($random list item(%CurrentUserAgentList))

The bot seems to take ages - it does not hang but just does not proceed.

I wanted to see how quick the random functions works on large lists.

Is there something wrong with the code or cant ubot handle such large lists?

I have attached the user agent files if anyone is looking for user agent lists.

Thanks for any feedback

do a search I made a snippet that works well for this but much smaller list.

JohnB · December 3, 2011

Try list from file instead of list from text...

John

mdc101 · December 3, 2011

HI

I have shrunk the file down to about 27 lines and it runs through pretty quickly.

What I did notice is it changes the useragent every 5th run.

Is this how the random list item works?

Is there a way to say grab 25 random urls from a large list text file and then have the used as the useragents and on every 5th run it grabs a new set of say 25 random urls?

Ideally one would think that random list item would be every run.

JohnB · December 3, 2011

Ok, the problem is the size of the list...if you watch this, I am using the whole list and the switch useragent is where it gets stuck:

http://screencast.com/t/I0Ig3Hsae

When I switched it to only use 300 of the list items it works fine:

http://screencast.com/t/hmcnXv4EH3

You will notice my setup in the video does not use random. I set up my own random function simply so I could control the size of the list without changing it. The entire list is loaded, but I am only using the first 300 items in the second video. It works just as well with $random list item. (which is the correct way to do it)

John

clear list(%useragents)

add list to list(%useragents, $list from file("{$special folder("Desktop")}\\user-agent-list.txt"), "Delete", "Global")

loop(25) {

set(#position, $rand(0, 299), "Global")

set user agent($list item(%useragents, #position))

navigate("http://whatsmyuseragent.com/", "Wait")

wait(4)

}

mdc101 · December 3, 2011

Ah very clever! Thanks John that's a cool way of working with a large list where you need randomness.

Thanks guys for the feedback, appreciated

Alliance · February 13, 2012

Hi I am new in this forum, and also in the Ubot world.

I am setting my first bot and discovered this ppost to be usefull for me, but unfortunatelly I can't download the useragent list attached.

Is there an special precedure to download files in this forum, it says forbitten acces for me, but I am logged in the forum.

Thanks for your help.

toothless · March 21, 2012

Hi I am new in this forum, and also in the Ubot world.

I am setting my first bot and discovered this ppost to be usefull for me, but unfortunatelly I can't download the useragent list attached.

Is there an special precedure to download files in this forum, it says forbitten acces for me, but I am logged in the forum.

Thanks for your help.

I have the same problem,What can I do also,very nice post

a2mateit · March 22, 2012

I have the same problem,What can I do also,very nice post

You need to PM a moderator to get your restriction lifted...

Website Rich · November 7, 2012

Why are all the attachments down on every post I visit? How do we download the ubots and txt files if they are all missing? Or is there a level you need to be before you're able to download any files? Please advise me on what I need to do. Thanks

Legend · November 7, 2012

Why are all the attachments down on every post I visit? How do we download the ubots and txt files if they are all missing? Or is there a level you need to be before you're able to download any files? Please advise me on what I need to do. Thanks

Are you registered for this forum? If not just send in a support ticket to axx the goodies... :rolleyes:

Twitter Dominator · November 8, 2012

im registered and their not loading for me either, but anyways the code is pasted for a copy paste in a few posts above so you dont need the lists to make it work you can scrape the useragent list from almost everywhere now, the code works in the abvove post

http://www.ubotstudio.com/forum/index.php?/topic/8641-swapping-user-agents/#entry44072

whoami · December 19, 2012

This is a good topic, too bad the user agent list he uploaded is not up anymore.

Anything we can do?

Code Docta (Nick C.) · January 21, 2013

Anyone have this list to re upload for us? would like to have!!

AutomationNinja · January 31, 2013

user_pref("useragentswitcher.1.appname", "");

user_pref("useragentswitcher.1.appversion", "");

user_pref("useragentswitcher.1.description", ":: BROWSERS - Windows ::");

user_pref("useragentswitcher.1.useragent", "");

user_pref("useragentswitcher.10.description", "Multizilla 1.6 (Win xp)");

user_pref("useragentswitcher.10.useragent", "Mozilla/5.0 (Windows; U; Windows XP) Gecko MultiZilla/1.6.1.0a");

user_pref("useragentswitcher.11.description", "Netscape 7.1 (Win 98)");

user_pref("useragentswitcher.11.useragent", "Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.4) Gecko Netscape/7.1 (ax)");

user_pref("useragentswitcher.12.description", "Netscape 4.8 (Win XP)");

user_pref("useragentswitcher.12.useragent", "Mozilla/4.8 [en] (Windows NT 5.1; U)");

user_pref("useragentswitcher.13.description", "Netscape 3.01 gold (Win 95)");

user_pref("useragentswitcher.13.useragent", "Mozilla/3.01Gold (Win95; I)");

user_pref("useragentswitcher.14.description", "Netscape 2.02 (Win 95)");

user_pref("useragentswitcher.14.useragent", "Mozilla/2.02E (Win95; U)");

user_pref("useragentswitcher.15.description", ":: SPIDERS - search ::");

user_pref("useragentswitcher.15.useragent", "");

user_pref("useragentswitcher.16.description", "Googlebot 2.1 (New version)");

user_pref("useragentswitcher.16.useragent", "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)");

user_pref("useragentswitcher.17.description", "Googlebot 2.1 (Older Version)");

user_pref("useragentswitcher.17.useragent", "Googlebot/2.1 (+http://www.googlebot.com/bot.html)");

user_pref("useragentswitcher.18.description", "Msnbot 1.0 (current version)");

user_pref("useragentswitcher.18.useragent", "msnbot/1.0 (+http://search.msn.com/msnbot.htm)");

user_pref("useragentswitcher.19.description", "Msnbot 0.11 (beta version)");

user_pref("useragentswitcher.19.useragent", "msnbot/0.11 (+http://search.msn.com/msnbot.htm)");

user_pref("useragentswitcher.2.appname", "Microsoft Internet Explorer");

user_pref("useragentswitcher.2.appversion", "4.0 (Compatible; MSIE 6.0; Windows NT 5.1)");

user_pref("useragentswitcher.2.description", "MSIE 6 (Win XP)");

user_pref("useragentswitcher.2.useragent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");

user_pref("useragentswitcher.20.description", "Yahoo Slurp");

user_pref("useragentswitcher.20.useragent", "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)");

user_pref("useragentswitcher.21.description", "Ask Jeeves/Teoma");

user_pref("useragentswitcher.21.useragent", "Mozilla/2.0 (compatible; Ask Jeeves/Teoma)");

user_pref("useragentswitcher.22.description", ":: BROWSERS - Mac ::");

user_pref("useragentswitcher.22.useragent", "");

user_pref("useragentswitcher.23.description", "Safari 125.8 (Mac OSX)");

user_pref("useragentswitcher.23.useragent", "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/125.2 (KHTML, like Gecko) Safari/125.8");

user_pref("useragentswitcher.24.description", "Safari 85 (Mac OSX)");

user_pref("useragentswitcher.24.useragent", "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/125.2 (KHTML, like Gecko) Safari/85.8");

user_pref("useragentswitcher.25.description", "MSIE 5.15 (Mac OS 9)");

user_pref("useragentswitcher.25.useragent", "Mozilla/4.0 (compatible; MSIE 5.15; Mac_PowerPC)");

user_pref("useragentswitcher.26.description", "Firefox 0.9 (Mac OSX )");

user_pref("useragentswitcher.26.useragent", "Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7a) Gecko/20040614 Firefox/0.9.0+");

user_pref("useragentswitcher.27.description", "Omniweb563 (Mac OSX)");

user_pref("useragentswitcher.27.useragent", "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-US) AppleWebKit/125.4 (KHTML, like Gecko, Safari) OmniWeb/v563.15");

user_pref("useragentswitcher.28.description", ":: BROWSERS - linux/unix/beos ::");

user_pref("useragentswitcher.28.useragent", "");

user_pref("useragentswitcher.29.description", "Epiphany 1.2 (Linux)");

user_pref("useragentswitcher.29.useragent", "ozilla/5.0 (X11; U; Linux; i686; en-US; rv:1.6) Gecko Epiphany/1.2.5");

user_pref("useragentswitcher.3.appname", "Microsoft Internet Explorer");

user_pref("useragentswitcher.3.appversion", "4.0 (compatible; MSIE 5.5; Windows NT 5.0)");

user_pref("useragentswitcher.3.description", "MSIE 5.5 (Win 2000)");

user_pref("useragentswitcher.3.useragent", "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0 )");

user_pref("useragentswitcher.30.description", "Epiphany 1.4 (Ubuntu)");

user_pref("useragentswitcher.30.useragent", "Mozilla/5.0 (X11; U; Linux i586; en-US; rv:1.7.3) Gecko/20040924 Epiphany/1.4.4 (Ubuntu)");

user_pref("useragentswitcher.31.description", "Firefox 0.8 (Linux)");

user_pref("useragentswitcher.31.useragent", "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040614 Firefox/0.8");

user_pref("useragentswitcher.32.description", "Galeon 1.3 (Linux)");

user_pref("useragentswitcher.32.useragent", "Mozilla/5.0 (X11; U; Linux; i686; en-US; rv:1.6) Gecko Galeon/1.3.14");

user_pref("useragentswitcher.33.description", "Konqueror 3 rc4 (Linux)");

user_pref("useragentswitcher.33.useragent", "Konqueror/3.0-rc4; (Konqueror/3.0-rc4; i686 Linux;;datecode)");

user_pref("useragentswitcher.34.description", "Konqueror 3.3 (Gentoo)");

user_pref("useragentswitcher.34.useragent", "Mozilla/5.0 (compatible; Konqueror/3.3; Linux 2.6.8-gentoo-r3; X11;");

user_pref("useragentswitcher.35.description", "Mozilla 1.6 (Debian)");

user_pref("useragentswitcher.35.useragent", "Mozilla/5.0 (X11; U; Linux; i686; en-US; rv:1.6) Gecko Debian/1.6-7");

user_pref("useragentswitcher.36.description", "Opera 7.23 (Linux)");

user_pref("useragentswitcher.36.useragent", "MSIE (MSIE 6.0; X11; Linux; i686) Opera 7.23");

user_pref("useragentswitcher.37.description", "ELinks 0.9.3 (Kanotix)");

user_pref("useragentswitcher.37.useragent", "ELinks/0.9.3 (textmode; Linux 2.6.9-kanotix-8 i686; 127x41)");

user_pref("useragentswitcher.38.description", "Elinks 0.4pre5 (Linux)");

user_pref("useragentswitcher.38.useragent", "ELinks (0.4pre5; Linux 2.6.10-ac7 i686; 80x33)");

user_pref("useragentswitcher.39.description", "Links 2.1 (Linux)");

user_pref("useragentswitcher.39.useragent", "Links (2.1pre15; Linux 2.4.26 i686; 158x61)");

user_pref("useragentswitcher.4.description", "MSIE 5.5 (Win ME)");

user_pref("useragentswitcher.4.useragent", "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)");

user_pref("useragentswitcher.40.description", "Links 0.9.1 (Linux)");

user_pref("useragentswitcher.40.useragent", "Links/0.9.1 (Linux 2.4.24; i386;)");

user_pref("useragentswitcher.41.description", "Lynx 2.8.5rel.1 (Linux)");

user_pref("useragentswitcher.41.useragent", "Lynx/2.8.5rel.1 libwww-FM/2.14 SSL-MM/1.4.1 GNUTLS/0.8.12");

user_pref("useragentswitcher.42.description", "w3m 0.5.1 (Linux)");

user_pref("useragentswitcher.42.useragent", "w3m/0.5.1");

user_pref("useragentswitcher.43.description", "Links 2.1 (FreeBSD)");

user_pref("useragentswitcher.43.useragent", "Links (2.1pre15; FreeBSD 5.3-RELEASE i386; 196x84)");

user_pref("useragentswitcher.44.description", "Mozilla 1.7 (FreeBSD)");

user_pref("useragentswitcher.44.useragent", "Mozilla/5.0 (X11; U; FreeBSD; i386; en-US; rv:1.7) Gecko");

user_pref("useragentswitcher.45.description", "Netscape 4.77 (Irix)");

user_pref("useragentswitcher.45.useragent", "Mozilla/4.77 [en] (X11; I; IRIX;64 6.5 IP30)");

user_pref("useragentswitcher.46.description", "Netscape 4.8 (SunOS)");

user_pref("useragentswitcher.46.useragent", "Mozilla/4.8 [en] (X11; U; SunOS; 5.7 sun4u)");

user_pref("useragentswitcher.47.description", "Net Positive 2.1 (BeOS)");

user_pref("useragentswitcher.47.useragent", "Mozilla/3.0 (compatible; NetPositive/2.1.1; BeOS)");

user_pref("useragentswitcher.48.description", ":: SPIDERS - misc ::");

user_pref("useragentswitcher.48.useragent", "");

user_pref("useragentswitcher.49.description", "download demon");

user_pref("useragentswitcher.49.useragent", "Download Demon/3.5.0.11");

user_pref("useragentswitcher.5.appname", "");

user_pref("useragentswitcher.5.appversion", "");

user_pref("useragentswitcher.5.description", "Avant Browser 1.2");

user_pref("useragentswitcher.5.useragent", "Avant Browser/1.2.789rel1 (http://www.avantbrowser.com)");

user_pref("useragentswitcher.50.description", "Email Wolf");

user_pref("useragentswitcher.50.useragent", "EmailWolf 1.00");

user_pref("useragentswitcher.51.description", "grub client");

user_pref("useragentswitcher.51.useragent", "grub-client-1.5.3; (grub-client-1.5.3; Crawl your own stuff with http://grub.org)");

user_pref("useragentswitcher.52.description", "gulperbot");

user_pref("useragentswitcher.52.useragent", "Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)");

user_pref("useragentswitcher.53.description", "ms url control");

user_pref("useragentswitcher.53.useragent", "Microsoft URL Control - 6.00.8862");

user_pref("useragentswitcher.54.description", "omni web");

user_pref("useragentswitcher.54.useragent", "OmniWeb/2.7-beta-3 OWF/1.0");

user_pref("useragentswitcher.55.description", "winHTTP");

user_pref("useragentswitcher.55.useragent", "SearchExpress");

user_pref("useragentswitcher.56.description", ":: v: 1.0.4 :: techpatterns.com ::");

user_pref("useragentswitcher.56.useragent", "");

user_pref("useragentswitcher.56.vendor", "http://techpatterns.com/forums/about304.html");

user_pref("useragentswitcher.6.appname", "Microsoft Internet Explorer");

user_pref("useragentswitcher.6.appversion", "4.0 (compatible; MSIE 6.0; Windows NT 5.0; en)");

user_pref("useragentswitcher.6.description", "Opera 8.0 (Win 2000)");

user_pref("useragentswitcher.6.useragent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; en) Opera 8.0");

user_pref("useragentswitcher.7.description", "Opera 7.51 (Win XP)");

user_pref("useragentswitcher.7.useragent", "Opera/7.51 (Windows NT 5.1; U) [en]");

user_pref("useragentswitcher.8.description", "Opera 7.5 (Win XP)");

user_pref("useragentswitcher.8.useragent", "Opera/7.50 (Windows XP; U)");

user_pref("useragentswitcher.9.description", "Opera 7.5 (Win ME)");

user_pref("useragentswitcher.9.useragent", "Opera/7.50 (Windows ME; U) [en]");

user_pref("useragentswitcher.import.overwrite", false);

user_pref("useragentswitcher.menu.hide", false);

user_pref("useragentswitcher.reset.onclose", true);

user_pref("useragentswitcher.user.agents.count", 56);

Swapping user agents

Recommended Posts

mdc101 15

Link to post

Share on other sites

Legend 181

Link to post

Share on other sites

Bob The Builder 62

Link to post

Share on other sites

JohnB 255

Link to post

Share on other sites

mdc101 15

Link to post

Share on other sites

JohnB 255

Link to post

Share on other sites

mdc101 15

Link to post

Share on other sites

Alliance 0

Link to post

Share on other sites

toothless 0

Link to post

Share on other sites

a2mateit 395

Link to post

Share on other sites

Website Rich 1

Link to post

Share on other sites

Legend 181

Link to post

Share on other sites

Twitter Dominator 125

Link to post

Share on other sites

whoami 26

Link to post

Share on other sites

Code Docta (Nick C.) 638

Link to post

Share on other sites

AutomationNinja 194

Link to post

Share on other sites

Join the conversation