mrvitej.in

Archive · Kubernetes

Back Up etcd Before You Upgrade

Snapshot cluster state, then upgrade a kubeadm cluster one minor version at a time.

1 min read

  • kubeadm 1.23
  • etcd 3.5
  • Kubernetes 1.23

All cluster state lives in etcd, so take a snapshot before any control plane change.

sudo ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%F).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
sudo ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-2022-04-04.db

Copy the file off the machine. Then upgrade, one minor version at a time (for example 1.22 to 1.23):

sudo kubeadm upgrade plan
sudo apt-get install -y kubeadm=1.23.5-00
sudo kubeadm upgrade apply v1.23.5
kubectl drain node1 --ignore-daemonsets
# upgrade kubelet and kubectl on the node, restart kubelet
kubectl uncordon node1

Upgrade control plane nodes first, then workers. Use PodDisruptionBudgets (GA since 1.21) so draining does not take down all replicas of a service.