Lets talk about scaling...

[Super Dave 26]

Hey guys I've been pretty out of tune with the forums lately but I hope we can get some good discussion from this one =] My director has a habit of poor planning and our team is normally tasked to put out the fires and 'just make it work' when things don't happen as he expects.

 

That being said our team built the backend architecture for ubots to check in and get job details, process those details depending on the type of bot, and finally report back with a status of the job.

 

For the individual bots themselves we started using virtual machinces to localize each session and cookie data but that didn't work for more than a few bots. From there we started acquiring more and more boxes at first there were 5, each running a native ubot and a virtual ubot. He wasn't happy with that so we got 5 more boxes and now everything is running just one bot.

 

That was chugging along nicely and our growth warranted more boxes and we got another 10, now giving us a total of 20 boxes. Something I should mention at this point the guy that handled most of the installs left our company. After that our director pretty much took 2 steps backwards hooked up monitors to all the boxes, mice, keyboards, the works. He really has no idea when it comes to technology these days /rant

 

Anyway, yesterday I came into the office to find these waiting for me..

 

http://i.imgur.com/wHY34l.jpg

Another 20 boxes.

 

After a few hours getting everything configured(and undoing a lot of what my director had done) I'm happy to say all of our boxes are up and running. woo! But the point of this post is about scaling. I've gone through a lot of headache over this and I'm hoping one of you guys out there know a thing or two about cluster computing, virtualizing, a more effective way to scale beyond this, or even a way to rework what I have currently to manage easier.

 

http://i.imgur.com/Xus6Sl.jpg

http://i.imgur.com/8GT9Vl.jpg

 

 

As I mentioned earlier our director is under the impression we need a keyboard mouse and monitor connected to each box but our team uses remote desktop to connect with each of the nodes in the event we need to do anything on the bots. How do you guys manage this type of thing?

[UBotBuddy 331]

The perfect solution is Outsourcing. LOL

 

Wow! I wish I was there. No a single keyboard and mouse there at the machines and Yes, do a remote to each box froim your desk.

 

Clustering...hmmmm.... I've never done that. But your setup looks nice. For what it's worth.

[UBotBuddy 331]

Get a Dropbox account for each of them and use one to build up to a nice size due to referrals. Give access to all of your machines to the master dropbox. Then you can share your .bot code easily betwwen machines.

 

What is the OS of each of the boxes? Win 7?

[Guerrilla 19]

Just enable multiple concurrent logons on each box and then run thr bot on as many accounts as the processor can handle. Resources are shared between accounts but cookies are not so this should work well fo you.

 

I would wager this would allow you to run many more bots per box as you are only loading the OS once. loading a seperate OS for each bot is overkill if its just cookies you want to localize.

 

You can then easily access each account / box from one pc with remote desktop.

 

Alternatively you might be able to use "sandboxie" to sandbox each bot so you can run all from the same account.

 

PS. LOL at all those keyboards, sort that **** out.

[meter 145]

You should look into Virtuozzo or VMWare ESXi (which is free iirc). Set up a Windows Server 2003 VM with .NET, Ubots, and Remote desktop on it configured. Then import them as many times as you like. That should be more than enough machines to suit your needs.

[JohnB 255]

I have a lot of respect for the knowledge you guys bring to the table! http://ubotstudio.com/forum/public/style_emoticons/default/rolleyes.gif

[Biks 9]

I was thinking of doing something like this on a much smaller scale. (2-3 computers) I've got a Twitter auto poster that I made in Ubot and I want to running 24/7. What's the cheapest "doo-hickey" that I need to switch keyboard/monitors between computers?

[UBotBuddy 331]

I use TightVNC for my computers in my home office. It's free.

[Super Dave 26]

Thanks everyone for your comments and suggestions. I've had some time to restructure our architecture and I've got a pretty good system that's working for the team and is super easy to manage. Here's how I did it..

 

After the post above I was told we'd be getting another 20 units to integrate into the system and I had to hurry to get a plan in place.

 

There'd be two Phases. First would be to deploy the new set then integrate that set into the bigger architecture when the whole thing gets restructured. So while I was working on the new boxes I had to name and configure the IPs on each machine to match to the box number. For example Box4 would have ip 192.168.1.104 and box 35 would end .135

 

Also I named each computer to match the box number as well. This helps on two levels - first, we can identify the box on the network by browsing the workgroup and selecting the box by number. Second, I have a script built into the bots to generate a unique string that's used as a session ID throughout the interactions that bot has with the rest of the infrastructure. The ID contains the PC Name which, now is the same as the box number and IP. Makes it real easy to track down bots through log files.

 

In order to deploy the boxes and maintain current campaigns I started off with mini clusters around the office...

 

This is a station from the pics above without the keyboards and other junk.

http://imgur.com/fXOri.jpg

 

I was limited with these clusters by the switches I had available but I managed to make due.

http://imgur.com/EqGsU.jpg

 

http://imgur.com/GeB0u.jpg

 

http://imgur.com/wy9Pn.jpg

 

http://imgur.com/pZygw.jpg

 

 

Once I had those clusters deployed we finished up the finial campaign before Phase Two.

 

First things first, Keyboards!! These three boxes are full and the two smaller boxes on the right are all our mice!

 

http://imgur.com/mDDas.jpg

 

Once we had all the keyboards out of the way I had room on the shelves to place the boxes. I had to break down all the clusters above and assimilate them into the new design. Now the battle was to maintain even power distribution on our breaker box. We had maintaince drop an additional line into each room from another breaker and this let us draw something like 150amps per room.

 

Box# 1-32

http://imgur.com/yU4Bd.jpg

 

From Behind

http://imgur.com/wq0QS.jpg

 

Box# 33-60

http://imgur.com/EioRZ.jpg

 

To manage them I've set up a console next to my desk that I can remote into from home. This console has a program called Remote Desktop Connection Manager and it gives me a nice interface to watch all the bots at once. In the screenshot below I have all the bots disconnected but you can see how they'd all display in a checkerboard pattern..

 

http://imgur.com/a5Dnvl.jpg

 

@BotBuddy - TightVNC was great. It worked well but I didn't like having to install extra software considering that MS provides remote desktop standard is it's OS distros.

 

@Biks - I'd say head over to bestbuy or someplace like that and get your standard switch/hub. Plug it into your router and set static IPs on the boxes. Make sure each PC has a username and password on it then you can use remote desktop to access them like I do. It's super easy to set up and maintain.

 

@meter - Thanks dude. I was trying hard to implement this but we don't have a PC we can dedicate to use with virtualizing at this time. I did look into it extensively and will be going that route if we need to expand past this point. Frankly we don't have the power (electricity) to handle many more boxes and will need the smaller footprint of a single box moving forward. Thanks for the tip

 

@Guerrilla - This would have been ideal and from what we've got set up right now it'll be an easy step to multiple concurrent logins. HOWEVER for some weird reason my bots don't actually click the "Tweet" button when it comes time to send. Everything else works as expected.. cookies, sessions, separate instances, etc.. the only thing that's not working(that i've noticed) is twitter not sending. Have you had this method work? Also, I tried sandboxie but couldn't get my bots to open. I looked it up here and come to find out there's a pretty funny topic about the error being a sandboxie issue. Get back to me if you have some suggestions about the tweet thing?

[Praney Behl 314]

Guys this is a very healthy discussion here, there is lot on this thread to learn.

[Biks 9]

What are the specs on those boxes? Are they all just running XP? I might have access to a bunch of older Dells (Pentium I's?) What's the slowest PC I would want to attempt to do that with? (Assuming I want to run a single Ubot on each and that's about it.)

[Guerrilla 19]

It might not be working because you have screensaver enabled. I found that when screensaver was enabled I had problems letting stuff run concurrently. This way worst case scenario you can leave the rdp connection open and it wont timeout and it should all work ok.

 

***EDIT***

 

I have just tried to test out exactly how to get this to work but for some reason compiled ubots are not working, I think its a problem ubot side. I'll have another go when issue is resolved.

[Guerrilla 19]

I sussed it!

 

My own work was proving boring so I decided to try and solve this problem.

 

Sandboxie indeed does not work. I did some tests and it looked like the problem was due to the fact that ubot makes calls to other apps (like for shell commands). Ubot and compiled bots do somehting like this when loaded, perhaps just checking things work.

 

The way I got it to work was really simple in the end. Just create multiple accounts on the box your working on and then right click on the bot while holding down the shift key.

 

This will bring up a menu option "run as". Just enter different account details for each bot you want to run.

 

Using this method you can now open up multiple instances of the bot without having to have multiple concurrent logons. This should get you the best performance possible.

 

I tested this with 2 instances of one of my bots running side by side and it worked perfectly. Cookies were not shared between the bots.

 

I am interested to see how many bots you manage to run concurrently. I am guessing a decent machine could probaly handle around 30 bots.

 

Also what I found interesting about this is that you can use a command line utility (like http://www.softtreetech.com/24x7/archive/53.htm) to do this for you. So this means you can create a master bot that with one click will open up as many instances as you like all running under different accounts.

 

If you keep the additional user account names and passwords the same for each box you can just copy this app to each box to make it super easy to load them all.

[Bob The Builder 62]

Hey guys I've been pretty out of tune with the forums lately but I hope we can get some good discussion from this one =] My director has a habit of poor planning and our team is normally tasked to put out the fires and 'just make it work' when things don't happen as he expects.

 

That being said our team built the backend architecture for ubots to check in and get job details, process those details depending on the type of bot, and finally report back with a status of the job.

 

For the individual bots themselves we started using virtual machinces to localize each session and cookie data but that didn't work for more than a few bots. From there we started acquiring more and more boxes at first there were 5, each running a native ubot and a virtual ubot. He wasn't happy with that so we got 5 more boxes and now everything is running just one bot.

 

That was chugging along nicely and our growth warranted more boxes and we got another 10, now giving us a total of 20 boxes. Something I should mention at this point the guy that handled most of the installs left our company. After that our director pretty much took 2 steps backwards hooked up monitors to all the boxes, mice, keyboards, the works. He really has no idea when it comes to technology these days /rant

 

Anyway, yesterday I came into the office to find these waiting for me..

 

http://i.imgur.com/wHY34l.jpg

Another 20 boxes.

 

After a few hours getting everything configured(and undoing a lot of what my director had done) I'm happy to say all of our boxes are up and running. woo! But the point of this post is about scaling. I've gone through a lot of headache over this and I'm hoping one of you guys out there know a thing or two about cluster computing, virtualizing, a more effective way to scale beyond this, or even a way to rework what I have currently to manage easier.

 

http://i.imgur.com/Xus6Sl.jpg

http://i.imgur.com/8GT9Vl.jpg

 

 

As I mentioned earlier our director is under the impression we need a keyboard mouse and monitor connected to each box but our team uses remote desktop to connect with each of the nodes in the event we need to do anything on the bots. How do you guys manage this type of thing?

 

Holy smokes, VMWare anyone?

[Bob The Builder 62]

I have a sneaky suspicion a $1000 server off Ebay (got my last 2xQuad 3.16 16GB Ram 6x146 SAS 15K system for $800 shipped, this thing rips and runs a ton of VMs) can probably run the load of every single one of those computers running the free version of ESXi (VMWare Enterprise Server). If it can't you can easily add another box and even go to the $500 license that gives you 3 servers and one control server. Life would be so much easier. This will even add in redundancy and real time failover if you had shared storage.

 

But with UBot 4, I don't understand why you can't use the same PC since it has it's own browser space.

 

I know this is an old post, you make any progress on this network?

[a2mateit 395]

Hoping to buy my own servers soon. Need to cut the ties now that T1 will soon be offered in my area.

 

Thinking of starting with one master server at first, then to eventually upgrade to five. Any recommendations or advice would be greatly appreciated, as I don't know much about this.

[Bob The Builder 62]

Hoping to buy my own servers soon. Need to cut the ties now that T1 will soon be offered in my area.

 

Thinking of starting with one master server at first, then to eventually upgrade to five. Any recommendations or advice would be greatly appreciated, as I don't know much about this.

 

You can get a lot out of one powerful server using VMWare and other solutions. VMWare ESXi 5 is completely free, if you want fail over and management tools you need to buy a license (starts at $500 and gets expensive fast). But even the free license you can run a few hundred machines off 2-3 servers pretty easily.

[blumi40 222]

>> you can run a few hundred machines off 2-3 servers pretty easily

never ever bob ! show me that

for sure on a doubleQuadCore and maximum ram u can run many instances of guests but 3 Hosts = ~ 100 ? no way i dont belive that.

[Bob The Builder 62]

>> you can run a few hundred machines off 2-3 servers pretty easily

never ever bob ! show me that

for sure on a doubleQuadCore and maximum ram u can run many instances of guests but 3 Hosts = ~ 100 ? no way i dont belive that.

 

I can easily get 10-30 VM's off my $800 servers I get. You can get servers that have more hardware and get about 4x the density from them (although the price goes up exponentially for those servers and are not commonly found used yet). I also don't think you really need one box as often as he is using them. So for 20 boxes, I'd suspect 5-10 VM's would be more than enough.

 

But 100 VM's for 3 servers is extremely doable although may not be ideal depending on the operating system and application requirements.

[a2mateit 395]

Hey Bob,

 

Sounds like the $800 servers you get are a great solution for me.

 

Mind PM'ing me the link, or you can post it up here.

 

It would be greatly appreciated.

[Bob The Builder 62]

Hey Bob,

 

Sounds like the $800 servers you get are a great solution for me.

 

Mind PM'ing me the link, or you can post it up here.

 

It would be greatly appreciated.

 

I just keep an eye out on Ebay for heavy servers for a good price.

I have some scripts I run that notify me as I always need hardware in my datacenter.

[a2mateit 395]

Can you suggest any specifications that I should look for, as I have almost zero knowledge on the subject.

[Bob The Builder 62]

Can you suggest any specifications that I should look for, as I have almost zero knowledge on the subject.

 

Lots of options there, but typically you can find a dual quad core with ~16GB ram loaded with SAS 10k (ideally 15K) drives for under $1,000 if you really keep your eyes peeled or have some automation in place. They worth twice the price but will handle a lot of virtual machines or one big machine if you go that route.

 

I have quite a few of them, I think I'm up to 12 or 13 right now.

[Anonym 53]

12-13 ? Holy cr..

 

I used to run a couple of ESX servers, back in V2.5. At that time it was mandatory to have SCSI disks.

Nowadays I actually prefer virtual servers on simpler desktop HW, quad core, 16 GB PM, a couple of network interfaces etc. I really like VirtualBox as virtualization software and it's free. I run WindowsXP, Windows2003 and Linux based servers (ok, WinXP is hardly a server OS, but sometimes it is needed... ) as virtual machines and it works great. On a 16GB host I usually run 3-4 hosts concurrently.

I do have an old Dell PowerEdge 2650, but the sounds from the fans are killing me!

 

Bob, it's very obvious to me that if you run your internet marketing business on that HW, then you are in another league than I am in.

 

I do have a server room with extra cooling in my house, so the geek factor is there to some extent, I guess.

[Bob The Builder 62]

12-13 ? Holy cr..

 

I used to run a couple of ESX servers, back in V2.5. At that time it was mandatory to have SCSI disks.

Nowadays I actually prefer virtual servers on simpler desktop HW, quad core, 16 GB PM, a couple of network interfaces etc. I really like VirtualBox as virtualization software and it's free. I run WindowsXP, Windows2003 and Linux based servers (ok, WinXP is hardly a server OS, but sometimes it is needed... ) as virtual machines and it works great. On a 16GB host I usually run 3-4 hosts concurrently.

I do have an old Dell PowerEdge 2650, but the sounds from the fans are killing me!

 

Bob, it's very obvious to me that if you run your internet marketing business on that HW, then you are in another league than I am in.

 

I do have a server room with extra cooling in my house, so the geek factor is there to some extent, I guess.

 

Hardware is cheap and doesn't call in sick. Just have to get what you want automated down and you are all set. A lot of it is research and pro-active tasks. Many of it is to avoid interactive requirements, so it is ready when I want it to be. Much of it is custom code, some now ubot, most of it is c# and perl, but doing more ubot & c# myself now.