Cluster Upgrade Link to heading

Well, that was interesting! Had a few tab completions fail and made me inquisitive. Turns out - mismatch between control plane and worker nodes AND kubectl client!

Gave the cluster the once over and found BREW overriding kubectl. What the heck is brew doing on the control plane? Must have been ssh into the node when I was working on devpod as I found mise had installed all my packages from config.toml there. The brew is on me though! 🤯 Smh…

Find below a nice upgrade path from 1.31 to latest current 1.34

kubectl is at 1.35. Happy Homelabbing folks! 🙂 🙃

Kubeadm Cluster Upgrade Guide Link to heading

Control Plane: v1.34.1 → v1.34.3 Link to heading

1. Pre-upgrade Checklist Link to heading

kubectl get nodes
kubectl get pods -A

sudo ETCDCTL_API=3 etcdctl snapshot save /tmp/etcd-backup.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

2. Update APT Repository Link to heading

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update

3. Upgrade kubeadm on First Control Plane Link to heading

sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm=1.34.3-1.1
sudo apt-mark hold kubeadm

kubeadm version
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.34.3

4. Upgrade kubelet and kubectl on First Control Plane Link to heading

kubectl drain <control-plane-node> --ignore-daemonsets --delete-emptydir-data

sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.34.3-1.1 kubectl=1.34.3-1.1
sudo apt-mark hold kubelet kubectl

sudo systemctl daemon-reload
sudo systemctl restart kubelet

kubectl uncordon <control-plane-node>

Worker Nodes: v1.31 → v1.34 Link to heading

Repeat this entire process for each worker node.

Step 1: Drain the Worker (from control plane) Link to heading

kubectl drain <worker-node> --ignore-daemonsets --delete-emptydir-data

Step 2: Upgrade to v1.32 (on worker node) Link to heading

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update

sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.32.11-1.1 kubectl=1.32.11-1.1
sudo apt-mark hold kubelet kubectl

sudo systemctl daemon-reload
sudo systemctl restart kubelet

Step 3: Upgrade to v1.33 (on worker node) Link to heading

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update

sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.33.1-1.1 kubectl=1.33.1-1.1
sudo apt-mark hold kubelet kubectl

sudo systemctl daemon-reload
sudo systemctl restart kubelet

Step 4: Upgrade to v1.34 (on worker node) Link to heading

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update

sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.34.3-1.1 kubectl=1.34.3-1.1
sudo apt-mark hold kubelet kubectl

sudo systemctl daemon-reload
sudo systemctl restart kubelet

Step 5: Uncordon the Worker (from control plane) Link to heading

kubectl uncordon <worker-node>
kubectl get nodes

Final Verification Link to heading

kubectl get nodes -o wide
kubectl get pods -n kube-system
kubectl cluster-info
kubectl version

# Test DNS
kubectl run test-dns --image=nicolaka/netshoot --rm -it --restart=Never -- nslookup kubernetes.default.svc.cluster.local 10.96.0.10

# Test Pod Scheduling
kubectl run test-nginx --image=nginx --restart=Never
kubectl get pod test-nginx -o wide
kubectl delete pod test-nginx

Notes Link to heading

  • Always upgrade one minor version at a time (e.g., 1.31 → 1.32 → 1.33 → 1.34)
  • Run apt-cache madison kubelet | grep 1.xx to find exact version strings
  • Worker nodes only need kubelet and kubectl upgraded (no kubeadm upgrade node)
  • Check for Homebrew kubectl conflicts: which kubectl 😄 ☸️