adrianhesketh.com

Immutable Windows Infrastructure on AWS

Immutable Infrastructure describes a situation where you don’t change infrastructure (usually servers) when they’re in-place, you just replace them with new ones. This normally goes hand-in-hand with automation of that work and provides a few benefits:

How does it work?

Auto Scaling Groups in AWS is a feature which (as the name suggests) allows you to configure rules which set when servers are added and removed to a pool. You can set it up to have minimums, maximums, change depending on how much load is hitting a load balancer, or just set to a schedule.

The machines started up by the Auto Scaling group all get configured to run a Powershell script (User Data in AWS terminology) to get themselves ready to receive traffic when they launch. At a minimum, that means installing some software to run, usually by downloading it from an S3 bucket, configuring logging etc.

It took me a few attempts to get the Powershell scripts right, but once it was working consistently, I was able to disable RDP access at the Security Group (firewall) level to prevent access to the boxes.

Here’s a sequence diagram of the process.

build_process

The Lambda which terminates old versions runs a Go program I wrote called the Terminator to do it: https://github.com/a-h/terminator

AWS Lambda doesn’t support Go programs directly, so I used a wrapper node.js wrapper to execute it:

The process is pretty straightforward and starts by logging to the build server and starting a deployment.

immutable_01

Selecting which environment you want to push to

immutable_02

Choosing which version to deploy:

immutable_03

An AWS Lambda is then triggered by the upload of the new deployment zip files to S3. This destroys a single instance in both the Web and API tiers, which we can see on our Grafana monitoring dashboards:

immutable_04

The Auto Scaling Group here is set to have 3 servers, so the healthy instance count should drop from 3 to 2 as the first server is destroyed.

This behaviour can be verified in the AWS Console (EC2 Auto Scaling Group). The Auto Scaling group logs the fact that an instance was automatically terminated, and a new instance is created to replace it:

immutable_05

The terminate_old_instances Lambda fires on a timer every 5 minutes. It collects the version number of all active instances and, if it finds old software running on an instance, it destroys the instance (to a configured minimum). Auto Scaling then kicks in again and creates a new instance running the latest version of the software.

To get this to work, I introduced a HTTP endpoint in all of the servers called “/version” which simply returns the version number of the running application.

The Lambda events are logged in CloudWatch:

2016-10-10T09:36:04.870Z    f5d15cb1-8ecc-11e6-94bc-5188e6ae168c    stdout: live_asg_web => lowest version 1.0.266, highest version 1.0.274, 2 instances to terminate
live_asg_web => terminating 2 of 3 instances
live_asg_web => terminating instance ids [i-01eb1b24b1e5d8a3e i-0ae0fbd754e6d71d9]

The example above shows that 2 instances are running 1.0.266 and one is running 1.0.274. The old instances can therefore be terminated and replaced with new instances running 1.0.274.

So our monitoring shows a server coming in and another one being destroyed.

immutable_06

As the new servers start up, they appear in the CPU graphs with higher CPU. This is expected, as starting up a .Net application results in JIT compilation and the creation of an IIS Application Pool process.

immutable_07

At the end of the release cycle, we’re back to 3 active nodes.

immutable_08

What I learned

There were a few things to know if you’re doing this yourself:

How do I get started?

I put together an example of how to use the Terminator as a Linux-based equivalent over at https://github.com/a-h/terminator-example