The myth about AWS and High Availability

Amazon loves talking about "high availability"; failover, health checks, disaster recovery, blah, blah blah. Well, there's actually an annoying problem they don't tell you about, and that is democracy. Hint, democracy involves the majority.

The problem is how do you maintain quorum-consensus across a cluster that spans only two availability zones when one of the zones goes out. The simple answer is, you can't.

Most clustering and fail over technologies rely on having the majority vote from the servers in the cluster before it will continue to function, otherwise known as quorum-consensus. This is to prevent a scenario often refereed to as split-brain. Clustered data stores that rely on such mechanism include Sql Server Always On Availability Group, MongoDB cluster, Elasticsearch Master Nodes and I'm sure there are many more.

If three nodes were to be partitioned into two availability zones, we actually see two outage scenarios. Either the majority nodes goes offline, or they don't.

It doesn't matter how many nodes you place on which availability zone either. If you have even number of nodes, they will all stop functioning when one AZ goes down. Or you have odd number of nodes, and you have 50% chance to stay up should one of the AZ goes down. What annoys me a bit is the fact that even AWS white paper on deploying Sql Server AOAG doesn't mention of this issue and recommend setting up file share witness on one of the DC that reside in the same AZ as one of the Sql Server instances.

Obviously not all regions are the same and some have three availability zones which are US East, US West (Oregon), Ireland, and Tokyo. The regions that have three AZ can comfortably distribute 1 nodes in each AZ. However, that's only 4 out of the 8 regions in total that can truly host HA data store with automatic fail over.

While this makes deployment of these data stores with high availability not as attractive, it's still very important to do so. Server restart on AWS is very real, and let's not forget these HA cluster will allow sysadmins to service the servers more easily without having to put up the "site down for maintenance" page.

One way to get around this issue is to establish a VPN connection to another data center or on premise servers and host a lightweight node outside of the AWS infrastructure. Sql Server AOAG has file share witness and MongoDB has arbiter nodes. These are nodes in the cluster that does nothing other than to cast vote and maintain quorum. It has also been mentioned that AWS VPC will in the future allow VPC peering across different regions, and that could also potentially solve this problem.

Or get a piece of wood and knock on it, hard.