[Kubernetes Lab 09] Diagnose Pod Failures and Drain a Node

Full series guide · Practice file ZIP

Windows 11 · VMware · Rocky Linux Kubernetes serial 09/10

A Pending Pod may have no container log yet, while CrashLoopBackOff calls for the previous exit reason and logs. Create separate failing workloads alongside the existing web application and use their states to choose the next diagnostic step. Finish by draining a worker and returning it to service.

1. Check nodes, pods, and events

kubectl get nodes -o wide
kubectl -n lab get deploy,pods,svc,pvc -o wide
kubectl -n lab get events --sort-by=.metadata.creationTimestamp

First, determine whether it is the entire node or limited to a specific namespace or Pod. Events disappear over time, and if the container changes, the log search method may also change. Leave the time of failure and the previous change together. If long-term archiving is required, set up a separate configuration to send Events and container logs to a central repository.

observation	main candidate	first check
Pending	Resource request·taint·volume·selection conditions	Events in describe pod
ImagePullBackOff	Image name, registry, authentication, communication	pull failure event
CrashLoopBackOff	Close app/Settings/Probe/OOM	logs –previous, lastState
Running but connection failed	readiness·Service selector·Policy·DNS	Communicate directly with EndpointSlice
NotReady	kubelet·runtime·CNI·disk pressure	describe node and node journal

2. Incorrect nodeSelector and Pending

pending-demo.yaml contains non-privileged conditions that can be created in a restricted environment. The reason the deployment fails is a non-existent node label. View events separately from resource shortages.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pending-demo
  namespace: lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pending-demo
  template:
    metadata:
      labels:
        app: pending-demo
    spec:
      automountServiceAccountToken: false
      nodeSelector:
        lab.maruops.com/pool: missing
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: sleeper
          image: busybox:1.37.0
          command: [sh, -c, "sleep 3600"]
          resources:
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              cpu: 100m
              memory: 64Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: [ALL]

kubectl apply -f ~/k8s-lab/app/pending-demo.yaml
kubectl -n lab get pods -l app=pending-demo
kubectl -n lab describe pod -l app=pending-demo

In the event, look for a statement that no nodes satisfy the node selector or affinity conditions. Since the execution node has not yet been determined, you should look at the deployment conditions before the app log. Here we only remove conditions that were intentionally placed. If your app requires an actual dedicated node, you should check your deployment design rather than removing the label.

kubectl -n lab patch deployment pending-demo --type=merge \
  -p '{"spec":{"template":{"spec":{"nodeSelector":null}}}}'
kubectl -n lab rollout status deployment/pending-demo --timeout=120s
kubectl -n lab delete deployment pending-demo

3. Reason for termination/previous log/disk

POD=$(kubectl -n lab get pods -l app=web -o jsonpath='{.items[0].metadata.name}')
kubectl -n lab describe pod "$POD"
kubectl -n lab logs "$POD" -c nginx --tail=100
kubectl -n lab get pod "$POD" -o jsonpath='{.status.containerStatuses[*].lastState}'
# 재시작 이력이 있는 컨테이너에만 이전 로그가 존재
# kubectl -n lab logs "$POD" -c nginx --previous --tail=100

Exit code 137 alone does not establish an OOM event. Check lastState reason, memory limits and the node OOM records together. If liveness triggered the restart, inspect the probe failure events. kubectl top requires metrics-server, which is not included in a default kubeadm installation. Distinguish a metrics API error from an application failure.

# 문제가 있는 해당 노드에서 실행
sudo journalctl -u kubelet -b --since '-30 min' --no-pager
sudo journalctl -u containerd -b --since '-30 min' --no-pager
sudo crictl ps -a
df -h
df -i

Even if there is space left, files cannot be created if the inode is full. Check whether logs, images, or volumes have increased and organize them. If you delete the entire /var/lib/containerd or /var/lib/etcd, you may lose the running status and recovery data.

4. PDB and drain

First confirm that the web Deployment has 2 replicas and both Pods are Ready. If the standalone volume-check Pod from Part 7 or the test Pod from Part 8 remains, remove it using the corresponding lab procedure. Save the following as web-pdb.yaml.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-budget
  namespace: lab
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: web

kubectl apply -f ~/k8s-lab/app/web-pdb.yaml
kubectl -n lab get pdb
kubectl -n lab get pods -o wide
kubectl drain wk01 --ignore-daemonsets --timeout=180s
kubectl -n lab get pods -o wide

drain removes nodes from new deployment targets and emits regular Pods to eviction. DaemonSet is left as --ignore-daemonsets. PDB helps ensure minimum usable numbers during these voluntary outages. It is not a function to prevent VM power off or disk failure, and the availability settings for rolling updates are also managed separately.

If a standalone Pod or emptyDir blocks the drain, check its owner and the purpose of its data first. Do not immediately add --force or --delete-emptydir-data. Local PV data does not automatically move to another worker and needs a separate plan. In this lab, clean up the standalone Pod from Part 7 before proceeding.

5. Node return and app response

# wk01에서 실행
sudo reboot

# cp01에서 wk01이 돌아온 뒤 실행
kubectl wait --for=condition=Ready node/wk01 --timeout=300s
kubectl uncordon wk01
kubectl get nodes
kubectl -n lab get pods -o wide

uncordon only allows new deployments, but does not automatically revert Pods that have been moved to other workers. If both workers are Ready, you can check the distribution by regenerating with kubectl -n lab rollout restart deployment/web when necessary. Verify node return, app readiness, and NodePort response from Windows, and complete the operation.

reference material

• Pod debugging procedure
• Pod life cycle and container status
• Safe drain of node
• PodDisruptionBudget
• Node compression and eviction

official documents and distribution files confirmed: October 18, 2026. The output is an example to determine completion and may vary depending on the environment.

---

← Part 08 · Part 10 →

Expand the table of contents for all 10 episodes

Full series guide and practice standards