I've spent a lot of my time recently working with containers and always find myself a bit frustrated using tools like Minikube because they feel like ‘easy-mode'. It feels a bit like cheating to me. I had wanted to build a HA kubernetes cluster for a whle, and after a number of unsuccessful attempt, I think I've finally cracked it, sort of.

 

I had tried on a number of occasions to follow various tutorials, official docs and so on, to help me build my own HA kubernetes cluster. None of these previous attempts went smoothly, and I either ended up giving up, or running out of time. It seemed to me that either the tutorials I was reading were out of date and steps had changed as the versions had evolved, or the documentation was missing important details or just plain wrong. Despite the inherent difficulty in deploying a HA kubernetes cluster, it still seems to be the most popular container orchestration platform around, so I thought, for the sake of science, this was something worth pursuing.

kubeadm

At the time of writing this, kubeadm doesn't provide native support for deploying HA clusters. There's an open issue on github which seems to indicate that this functionality will be being added at some point in the near future; (https://github.com/kubernetes/kubeadm/issues/261). For now though, there are a bunch of manual steps involved in using kubeadm to set up a HA cluster.

Other Deployment Tools

I'm aware that there are other tools out there which can deploy a HA kubernetes cluster for you, like Kismatic and Rancher, but I wanted to see if I could do it myself without the help of such tools.

vagrant-ha-kubernetes

I've created a repo on github (https://github.com/llharris/vagrant-ha-kubernetes.git) which aims to make the whole process a little bit easier. This is very much a first draft, and has a LOT of room for improvement, but it seems to work quite well. The repo provides a Vagrantfile which will spin up a 6 node cluster (3 x master, 3 x worker) using the Virtualbox provider. It then uses the vagrant ansible_local provisioner and a bunch of playbooks to perform all of the pre-requisite steps needed to form the HA cluster.

Vagrant Sequential Actions

The last few steps aren't automated however and the actual process of standing up the cluster with kubeadm init and joining workers is still manual. The reason for this is due to the sequential way that vagrant deploys guests and runs provisioner scripts or playbooks.

Before kubeadm init will work properly on the first master node, etcd needs to be up and running on all 3 of the master nodes. This means we need to start etcd on masters 1,2 and 3, then run kubeadm init on master 1 before going back to master 2 and 3 to perform some more steps. Vagrant however performs all the steps sequentially so it will try to run everything it needs to against master1 before moving onto master2 and then master3. There might be a way around this, but if there is, I'm not sure what.

ansible_server provisioner

The only other way to approach this AFAIK would be to use the ansible_server provisioner, let vagrant deploy all of the guest VMs, then on the last VM run all the playbooks from there (as an ansible server) and carry out all the steps in one shot where we can control which VMs are acted upon and in which order.  This is something I'd like to explore, so it's on the to-do list for now.

Usage

If you want to give this a go, you'll need a machine with at least 16GB of RAM (more = better). Head over to the github repo and follow the instructions there. It has only been tested on my PC running Windows 10, with specific versions of Virtualbox and vagrant as outlined in the github README, so YMMV.

Kubernetes Dashboard

After you've got your HA cluster up and running, you'll probably want to be able to use the graphical kubernetes dashboard. My friend José Gomez has an excellent article on his blog which explains the correct way to do this. Check it out: http://www.joseluisgomez.com/containers/kubernetes-dashboard/