Have you noticed that many people operating Kubernetes clusters recommend replacing the cluster rather than upgrading it in place?
Reading Time: 2.5 minutes
That is, to upgrade from Kubernetes 1.11 to 1.14 you:
- deploy a new 1.14 cluster
- test the new cluster meets your applications’ very high expectations
- migrate applications to the new cluster
- decommission the 1.11 cluster
This looks like a blue-green release pattern, but why? Surely this is more work than upgrading in-place and requires additional hardware capacity.
The Kubernetes version and version-skew policy offers some clues:
Kubernetes versions are expressed as x.y.z, where x is the major version, y is the minor version, and z is the patch version, following Semantic Versioning terminology.
So the Kubernetes 1.11 or 1.14 releases offered as managed services and supported by tools are minor releases.
Arguably, Kubernetes does not follow the semantic versioning standard because it has been removing resource APIs. SemVer only permits additions and bugfixes in minor and patch changes. Though you could also argue the production version of core Kubernetes resource APIs hadn’t been published yet.
¯\_(ツ)_/¯
Either way, better read those release notes carefully and test with your deployment and software stack.
Minor releases occur approximately every 3 months, so each minor release branch is maintained for approximately 9 months.
So Kubernetes changes very quickly. Release branches are maintained for only 9 months, and major Cloud providers and many tools only support maintained releases. Must upgrade if support is important.
Project policies for API deprecation and API change guidelines require kube-apiserver to not skip minor versions when upgrading, even in single-instance clusters
So kube-apiserver must be upgraded through each release . That is, the cluster control plane upgrade path from 1.11 to 1.14 must transit 1.11->1.12->1.13->1.14.
You cannot skip from 1.11 to 1.14.
In highly-available (HA) clusters, the newest and oldest kube-apiserver instances must be within one minor version.
So gotta upgrade all the kube-apiserver instances, not just one.
kubectl is supported within one minor version (older or newer) of kube-apiserver.
So the command-line tool to manage cluster resources and applications running on the cluster needs to update in-step with the apiserver. Good thing kubectl is a static golang binary and easy to vendor according to version.
kubelet must not be newer than kube-apiserver, and may be up to two minor versions older.
So all the of worker nodes must update to at least one of the intermediate releases between 1.11 and 1.14, 1.12 or 1.13, and then upgrade again to 1.14. If you have 100 worker nodes in the cluster, that’s 200 kubelet upgrade operations that must work correctly.
I understand that you’re a smart, capable engineer and could automate and test these upgrades. But is it a good use of your time and energy?
Takeaways
You’ll probably want to update each cluster to a/the recent release once every six months to stay comfortably within the nine-month maintained release window.
Many of the arguments applied to replacing virtual machines rather than changing them have apply to Kubernetes clusters.
Disposable infrastructure never looked so good.
Provisioning a new cluster with the fresh software and migrating applications to it one-by-one may eliminate a lot of busy work and risk from bugs that only occur with some rare sequence of events. You also retain the option to fallback application deployments to the old cluster.
Automated testing of your Kubernetes-based compute platform, managed Kubernetes services (EKS, GKE, AKS), and Kubernetes management tooling (Rancher) are looking a whole lot better.
Stephen
#NoDrama
