Utilize the Swarm: An approach for rolling out web apps with Virtual Machines and a bit less pain

(Picture of a simplified Dyson swarm)

To be clear, I'm still working out the kinks in this particular plan and it may work better for people in a configuration similar to the one we have at the university I work at. The reason being we can change which computer (or in this case VM) a host name points to by simply changing the MAC address of the host in a central database, or by moving an alias as we so choose. This makes changing targets pretty straight forward, but should still be pretty easy with a little help and co-operation from your local network admins if you don't have a similar system in place.

This will take a bit of setup before your ready to roll, and may simply not work in some environments dependent on your specific IT infrastructure. This article will be written around using VMware's esx/server 2 technology + VMware Workstation. The idea in general that I'm alluding to here as 'the swarm' is to create a central (easily replaceable) system that simply points at your 'swarm' of virtual machine applications.

This dodges a few sticky points of rolling out a new service, or alternatively updating older services as it eliminates the chance of differences between the development server and the production server for a given application.

Depending on your environment and work method the initial setup is going to be a bit different. Our group tends to have individual projects that are end to end developed, documented and rolled out by a single author. As such we can publish new VM's using VMWare Workstation and having it stored locally on our system while we set it up and tune it. If you are in a collaborative environment either a 'development' server that holds current VM's or just your VM server will do nicely for this portion of the task. You can also use some of the more extensive tools available from VMWare like VM Motion to make things even easier.

The next thing you are going to need is a way to proxy your internal servers. We utilize Apache’s proxy system for this purpose, but you could use any of a number of solutions including a hardware load balancer I would imagine. You can even use apache to perform a load balanced proxy which is incredibly useful for serving out a horde processes of the same type to the wider world seamlessly. For instance you can have a group of ruby server processes stored on 3 separate systems all pointed too by the web path /ruby-site/. However I’ve run into some issues with delivery of public resources (images and css for example) as they are referred to by relative pathing in many applications and this will no longer work. That relative pathing is broken unless your mapping between the apache proxy and your internal server’s configs is done extremely carefully. An easy out for this is to utilize resource servers that serve out that information separately and refer to the resources by their resource servers’ direct paths (this resource server could even be the same underlying server, accessed via its own name instead of the proxy). Configuring around host names and IP addresses each has its own advantages in my opinion, but it really only matters where / how you’re given IT structure manages IP addresses. In our climate we can use hostnames and alias host names, so it’s trivial to simply use an alias as the target host. Then all you have to do is a simple database update as to where that alias points and your done, nothing ever gets touched on the front end, one minute it was pointing at box X, then it’s pointing at box Y.

Once the proxy system is configured you need to point it at the appropriate application servers for the portions of your website you wish to publish currently. This is where this approach tends to be more helpful than the standard rollout method as you don’t have to actually make any changes on your ‘front end’ server. Instead as stated above once the new version of the page is up and running satisfactorily and executing in production mode instead of development / testing mode you can simply change your alias target and the new page is ‘published’. This also means that if it turns out that you have to roll back to a previous version of the website you simply bring up that VM and move the alias back.

The Blue Pill:

There are several major pitfalls to this current version of the idea, as this is just a rough outline of an overall infrastructure plan I’ve been kicking around that’s not terribly surprising. The most annoying issue I can think of is the resource handling; this method introduces a necessary level of indirection for the public resources to keep everything clean. This doesn’t have to be bad, especially if you have some sort of mega parallel system for delivering resources, then you’re probably doing this already for that portion of the site. Also, setting up a two way proxy that passes things like java script session data thru to the underlying web servers is not as trivial as it really should be (at least in apache).

The Red Pill:

There are a number of good things to say about this as well, but it is important to remember that even this idea which would arguably make it much easier to roll out new versions of a website can’t be done without at least a little bit of careful planning. That said this beats the hell out of waking up at 2 am to copy files over to the server when nobody is (in theory) using it so as to do a rollout under the radar of the user base. Instead you can have it up and confirmed well before hand, and simply toggle the alias during your downtime window (then clean up anything you may have forgotten). It also lets you store a snapshot of your entire web server including its full operating environment if you need to debug it on a large scale or work on it out of band of the executing environment (to do performance tuning for example).

As always, opinions and critiques are highly welcome and appreciated.