[Kubernetes Lab 10] Restore etcd and Application Files; Apply Patch Upgrades

Full series guide · Practice file ZIP

Windows 11 · VMware · Rocky Linux Kubernetes serial 10/10

etcd state and application files need separate backups and recovery procedures. Using cp01, wk01 and wk02 from the lab, this part walks through a single-control-plane etcd restore and local application data recovery. It then covers the patch upgrade sequence within the same Kubernetes minor release.

1. Backup scope of etcd and app files

target	Storage contents	Something that just can’t be repaired
etcd snapshot	Kubernetes API objects and cluster state	Actual files in PV/external DB
/etc/kubernetes	PKI·kubeconfig·static Pod settings	Walker’s app data
Git/practice manifest	Desired declared state for apps and policies	Data changed during execution
local data of wk01	7 history.log, etc.	Cluster certificate/API object

One cp01 is responsible for both etcd and the API server, so if this VM stops, control tasks such as new deployments and scheduling will also stop. Even if some containers in a worker continue to run, it does not mean that the control plane is healthy. If availability is required, multiple control planes, etcd quorum, load balancers, and fault domains must be designed. The limit of three VMs on one Windows PC also remains.

2. Create cp01 etcd snapshot

Run from the labadmin shell on cp01. Reduce client version differences with etcdctl included in the etcd container. Create a snapshot briefly on the host mount path and then move it to the protected backup directory. Do not use copies of existing data files as snapshots.

BACKUP="/var/backups/k8s/$(date +%Y%m%d-%H%M%S)"
sudo install -d -m 0700 "$BACKUP"
ETCD_POD=etcd-cp01
ETCD_IMAGE=$(kubectl -n kube-system get pod "$ETCD_POD" \
  -o jsonpath='{.spec.containers[0].image}')
kubectl -n kube-system exec "$ETCD_POD" -- etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  snapshot save /var/lib/etcd/lab-snapshot.db
sudo mv /var/lib/etcd/lab-snapshot.db "$BACKUP/etcd.db"
sudo chmod 0600 "$BACKUP/etcd.db"
printf '%s
' "$ETCD_IMAGE" | sudo tee "$BACKUP/etcd-image.txt" > /dev/null
sudo tar -C / -czf "$BACKUP/etc-kubernetes.tgz" etc/kubernetes
sudo chmod 0600 "$BACKUP/etc-kubernetes.tgz" "$BACKUP/etcd-image.txt"

etcd snapshots contain sensitive state such as Secrets, and PKI backups contain private keys. Encryption and access restrictions are applied and stored in separate storage outside the VM. Set a retention cycle and don’t place it in your WordPress upload folder or public Git. If you use storage encryption, prepare recovery of encryption keys and settings separately.

3. Check snapshot status/hash

sudo ctr -n k8s.io run --rm \
  --mount "type=bind,src=$BACKUP,dst=/backup,options=rbind:ro" \
  "$ETCD_IMAGE" lab-etcd-check \
  /usr/local/bin/etcdutl snapshot status /backup/etcd.db --write-out=table
sudo sha256sum "$BACKUP/etcd.db" "$BACKUP/etc-kubernetes.tgz" \
  | sudo tee "$BACKUP/SHA256SUMS" > /dev/null

Check if the revision, number of keys, size, etc. are displayed. Hashing is intended to verify changes during transmission and is not a substitute for recovery testing. ctr -n k8s.io uses CRI’s containerd namespace. If the etcdutl path or inclusion in the image is different, check the official tool for the etcd version before proceeding.

4. Single etcd restoration conditions and procedures

The restoration below is for the single etcd member cp01 in this series, and applies only when the original IP/hostname and valid /etc/kubernetes PKI are still intact. This is different in scope from disaster recovery, which involves creating a new entire VM, or member replacement procedures in multi-member etcd. Do not turn on the clone lab with the same name and IP as the original VM on the same network at the same time. Keep a backup outside of the VM, create a revert point for practice, and then proceed.

It is easy to check the point in time when you restore an object created after a snapshot and then look it up. Create a test ConfigMap on cp01 and specify the actual backup path. Also compare the name and peer URL of the saved etcd manifest with the values ​​below.

kubectl -n lab create configmap after-snapshot --from-literal=created=after-backup
# 실제 생성된 디렉터리 이름으로 변경
RESTORE_BACKUP=/var/backups/k8s/20261018-120000
sudo test -s "$RESTORE_BACKUP/etcd.db"
sudo grep -E -- '--name=|--initial-cluster=|--initial-advertise-peer-urls=' \
  /etc/kubernetes/manifests/etcd.yaml

After taking a snapshot, you must have created a ConfigMap. If the test in the above path fails, the next command will not proceed. The Kubernetes API will be interrupted during the operations below. The original data directory is kept instead of immediately deleted, and static Pod manifests are moved out of the watch directory. If you leave it as a .bak file in the same directory, the kubelet can read it.

HOLD=/root/k8s-manifests-hold
sudo install -d -m 0700 "$HOLD"
sudo mv /etc/kubernetes/manifests/*.yaml "$HOLD/"
# kubelet이 static Pod를 종료할 때까지 반복 확인
sudo crictl ps

After confirming that etcd, kube-apiserver, kube-controller-manager, and kube-scheduler are all gone in crictl ps, proceed to the next step. The order is important because stopping the kubelet first may leave behind existing static Pod containers. Keep the containerd service running as it is required for the restore tool to run.

sudo systemctl stop kubelet
ETCD_IMAGE=$(sudo cat "$RESTORE_BACKUP/etcd-image.txt")
OLD_ETCD="/var/lib/etcd.pre-restore-$(date +%Y%m%d-%H%M%S)"
sudo mv /var/lib/etcd "$OLD_ETCD"
RESTORE_PARENT=$(sudo mktemp -d /var/lib/etcd-restore.XXXXXX)
sudo ctr -n k8s.io run --rm \
  --mount "type=bind,src=$RESTORE_BACKUP,dst=/backup,options=rbind:ro" \
  --mount "type=bind,src=$RESTORE_PARENT,dst=/restore,options=rbind:rw" \
  "$ETCD_IMAGE" lab-etcd-restore \
  /usr/local/bin/etcdutl snapshot restore /backup/etcd.db \
  --data-dir=/restore/data \
  --name=cp01 \
  --initial-cluster=cp01=https://192.168.80.10:2380 \
  --initial-advertise-peer-urls=https://192.168.80.10:2380 \
  --initial-cluster-token=lab-restore-20261018 \
  --bump-revision=1000000000 --mark-compacted

Proceed only after confirming the success of the restore command. If you go back to a past revision, the Kubernetes watch cache can retain previous information. Raise the revision and mark the previous section as compacted to force the client to reread the state. This number is not the amount of data recovered or the retention period.

sudo test -d "$RESTORE_PARENT/data/member"
sudo mv "$RESTORE_PARENT/data" /var/lib/etcd
sudo mv "$HOLD"/*.yaml /etc/kubernetes/manifests/
sudo systemctl start kubelet
# API 서버 기동 후 확인
kubectl get --raw='/readyz?verbose'
kubectl get nodes
kubectl -n lab get deployment,pods,svc,pvc
kubectl -n lab get configmap after-snapshot

The last ConfigMap lookup should be NotFound. A connection rejection or authentication failure is not considered a “success because the object is gone.” The restoration is complete only when the API, node, and actual web response are normal. In case of failure, check the kubelet·etcd logs, preserving the old data directory and original manifest. Do not blindly repeat the same restore command.

5. Local PV file backup/restore

Delete the volume-check Pod to back up wk01’s data without the app writing any files. Real DBs use consistent backup or write-stop procedures provided by the engine. Transactional consistency cannot be guaranteed by running tar alone.

# wk01에서 실행
sudo install -d -m 0700 /var/backups/k8s
sudo tar -C /var/local/k8s-lab -czf /var/backups/k8s/local-data.tgz data
sudo chmod 0600 /var/backups/k8s/local-data.tgz
# 복구 시험 전, 이 파일도 VM 밖으로 보관
# 기존 디렉터리를 보관하고 백업을 같은 위치로 복원
sudo mv /var/local/k8s-lab/data \
  "/var/local/k8s-lab/data.before-restore-$(date +%Y%m%d-%H%M%S)"
sudo tar -C /var/local/k8s-lab -xzf /var/backups/k8s/local-data.tgz
sudo ls -ld /var/local/k8s-lab/data

# cp01에서 복구 파일 확인 후 정리
kubectl apply -f ~/k8s-lab/app/volume-check.yaml
kubectl -n lab wait --for=condition=Ready pod/volume-check --timeout=120s
kubectl -n lab logs volume-check
kubectl -n lab delete pod volume-check

Check previous records and owner UID/GID. Also check the timing of etcd and file backups. Avoid situations where only the PVC is recovered but the file is missing, or the file is linked to another app’s PVC.

6. Patch changes in the same minor series

The procedure below moves to a newer patch within the 1.36 series. The latest repository version at the original check date was 1.36.5; if that version is already installed, verify it and wait for a newer patch before performing an upgrade. Review CNI and runtime compatibility and release notes first. Move through minor releases one at a time; changing a version number does not make it safe to skip several minor releases.

# cp01에서 현재 버전과 저장소 확인
kubectl get nodes
sudo dnf --disableexcludes=kubernetes list --showduplicates kubeadm
sudo kubeadm upgrade plan

After checking the backups and app status, I’ll target the newer 1.36 patch when it’s actually available. 1.36.5 below is an example variable format. If you already have the same version, skip this change step. Specify TARGET again in the new shell and on each node as well.

# cp01: 실제 대상 패치로 변경한 뒤에만 실행
TARGET=1.36.5
sudo dnf install -y "kubeadm-${TARGET}" --disableexcludes=kubernetes
sudo kubeadm upgrade apply "v${TARGET}"
kubectl drain cp01 --ignore-daemonsets --timeout=180s
sudo dnf install -y "kubelet-${TARGET}" "kubectl-${TARGET}" --disableexcludes=kubernetes
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl wait --for=condition=Ready node/cp01 --timeout=300s
kubectl uncordon cp01

Then upgrade wk01 and wk02 one at a time. Do not start the next worker until the first worker is Ready again and the application is healthy. Run each worker command on the target worker; run drain and uncordon on cp01.

# cp01에서 먼저
kubectl drain wk01 --ignore-daemonsets --timeout=180s

# wk01에서: TARGET은 cp01과 같은 실제 대상 버전
TARGET=1.36.5
sudo dnf install -y "kubeadm-${TARGET}" --disableexcludes=kubernetes
sudo kubeadm upgrade node
sudo dnf install -y "kubelet-${TARGET}" "kubectl-${TARGET}" --disableexcludes=kubernetes
sudo systemctl daemon-reload
sudo systemctl restart kubelet

# cp01에서 복귀 확인. 이후 wk02에도 같은 순서를 적용
kubectl wait --for=condition=Ready node/wk01 --timeout=300s
kubectl uncordon wk01
kubectl get nodes -o wide
kubectl -n lab rollout status deployment/web --timeout=180s

Replacement of containerd is carried out separately from the Kubernetes upgrade. We intentionally release the versionlock, select a compatible version, and then perform drain, verification, and refreezing for each node. It is easy to find the cause of failure if you do not change multiple layers at once. When making minor changes, the series URL of kubernetes.repo is also changed according to the official procedure.

7. Certificate/node/backup inspection records

kubectl get nodes
kubectl get pods -A
kubectl -n lab get deployment,pdb,pvc
sudo kubeadm certs check-expiration

Record certificate expiration date, disk/inode, restart increment, backup time, and most recent restore test date. After renewing the certificate, restarting the static Pod and deploying the user kubeconfig are confirmed through official procedures. Decide not only whether routine commands will run, but also who will check for failures and when.

If you have followed this far, let us explain again how to configure a new cluster, request path, node check, and restore status and files. You can use the previous exercise to double-check for blockages. Afterwards, you can expand based on what you currently need: multiple control planes, CSI, TLS gateway, and centralized logs and metrics.

reference material

• etcd 3.6 Snapshot and recovery/revision processing
• etcd backup/operation guide
• kubeadm upgrade sequence
• Kubernetes Version Differences Support Policy
• kubeadm Certificate Management
• Stored data encryption

official documents and distribution files confirmed: October 18, 2026. The output is an example to determine completion and may vary depending on the environment.

---

← Part 09

Expand the table of contents for all 10 episodes

Full series guide and practice standards