Transcription
Let's see one Kubernetes interview question. Your Kubernetes production cluster has 40 nodes and it's running a Kubernetes version that's about to lose support. You need to upgrade the whole cluster, but this cluster receives massive traffic 24/7 and you can't afford to let this cluster go down. How will you upgrade it? You have to make the change in stages.
A cluster has two halves, control plane nodes and worker nodes. The control plane is the brain and it has the API server, the scheduler, the parts that decide where pods run. The worker nodes are the muscle. These are the machines where pods actually run. And the important point we should always remember is that your pods do not need the control plane alive to keep serving traffic. If the control plane goes quiet for a minute, running pods keep serving traffic and users notice nothing.
So, in stage one, we will upgrade the control plane first. And while the control plane upgrades, your worker nodes keep serving traffic. You should also know that Kubernetes allows worker nodes to have an older version than the control plane, but never a newer version.
After upgrading the control plane in stage two, we will upgrade the worker nodes one node at a time. Let's see one worker node in action. First, you will cordon the node. That means the scheduler stops placing new pods on this node, but the pods already there keep running. Then, you will drain the node. Kubernetes removes running pods gently, one by one, and reschedules each of these pods onto the other 39 nodes. Once the node is empty, we will upgrade it to the new version. Then, uncordon it so that pods can come back.
Before draining a node, you also need to protect the applications running on this node. Say your checkout application has three identical pods and two of them happen to be on the node you're draining. If Kubernetes removes both pods at once, only one checkout pod remains to handle all the traffic. To prevent that, we create a pod disruption budget. It is is rule that says at least two checkout pods must remain healthy during planned maintenance. So, Kubernetes removes one pod, waits for its replacement to become healthy on another node, and only then removes the next one. The upgrade takes longer, but the application stays available.
Stage three, repeat the process 39 more times. Once you trust the process, you can do small batches as long as the remaining nodes have room to absorb the deleted pods. This is also why managed Kubernetes services do a surge upgrade. Add a brand new node running the new version first, then drain an old one, so total capacity never dips.