MaruOpsInfrastructure and AI notes

About (Korean)Contact (Korean)

Kubernetes cgroup v2 Migration: Avoid a kubelet Startup Failure After Upgrade

Server racks and overhead cable trays at NERSC

Reference photo · Server racks and overhead cable trays at NERSC · Photo: Derrick Coetzee / Source / CC0 1.0 · Resized for listings; illustrative photo, not the tested environment

The Kubernetes project published guidance on cgroup v1 deprecation and migration to v2. Linux cgroups control resource usage. The article highlights that kubelet does not start on a cgroup v1 node by default from Kubernetes 1.35. Kubernetes official announcement

A cluster can contain nodes with different OS images and runtime settings. Record each node’s OS, kernel, runtime, and cgroup configuration rather than tracking only the control-plane version.

On this page

Resource declarations depend on the node’s kernel configuration

CPU and memory requests and limits are Kubernetes declarations, but Linux kernel mechanisms enforce resource accounting and isolation for the actual processes. Cgroups provide that grouping and control layer. The kubelet and container runtime must manage it consistently. Updating the control-plane version does not by itself change the resource-control mode of every node in the cluster.

Cgroup v2 provides a unified hierarchy and resource-control interface. Support in an operating system is different from the mode active on a booted node. Earlier images or boot settings can leave a different mode in use. Mixed node pools therefore require checking the actual state of each image and configuration instead of inferring it from the distribution name.

A kubelet startup failure precedes application-level troubleshooting

The kubelet manages Pod execution on a node. If it fails to start after an upgrade, application logs alone will not explain the failure. Compare the described v1.35 defaults and preflight checks with the node logs, then distinguish a cgroup rejection from other configuration errors. Any temporary override needs an explicit reason, owner, and removal plan.

Cgroup version and cgroup driver are separate settings. The driver describes how the kubelet and runtime manage the resource hierarchy, and their configurations must be compatible. Finding a v2 filesystem does not complete the runtime-and-driver review. Maintain node images against the supported operating-system, kernel, and runtime combination rather than validating each component in isolation.

Compare behavior while converting one node pool

Start with a read-only filesystem check on a representative host. The command below returning cgroup2fs indicates a cgroup v2 filesystem mounted at that path. A different result needs further investigation of the host’s boot and mount configuration. A view from inside an arbitrary container should not be treated as proof of the configuration of the whole host.

Before conversion, define spare capacity, workload movement, and stop conditions. Afterward, compare Pod startup, memory accounting, OOM and eviction records, and monitoring collection with the earlier state. Agents or scripts reading cgroup files directly deserve particular attention. Promote the validated node image to subsequent pools only after those operational checks succeed.

stat -fc %T /sys/fs/cgroup

cgroup2fs describes this mount point. Review kubelet configuration and runtime driver compatibility separately. No actual node output or successful migration is claimed here.

Start with one node

  • Inspect the host’s cgroup filesystem and runtime driver.
  • Compare kubelet and kubeadm preflight requirements for the target upgrade.
  • Drain and test nodes individually before continuing the rollout.

Do not let a temporary exception become an undocumented permanent configuration. Record affected nodes and an exit plan before the next upgrade.

Announcement reference date: 2026-10-06 · Checked: October 24, 2026