For a Pod stopped at ImagePullBackOff, kubectl logs may have no application output because the image pull failed before the program started. Read the latest failure in the Pod’s Events first. Use that message to distinguish an image reference problem from authentication, network or registry errors.
The example below assumes a Kubernetes cluster with access. Replace demo with the namespace, web with the deployment and container name, and POD_NAME with the actual Pod name. The private registry address registry.example.com is also an example for illustrative purposes only and is not the server you will actually connect to.
On this page
Determine what the Pod wants to receive from which node
kubectl get pods -n demo -o wide
kubectl describe pod POD_NAME -n demo
kubectl get events -n demo \
--field-selector involvedObject.name=POD_NAME \
--sort-by=.metadata.creationTimestampImagePullBackOff is failing to retrieve the image and is increasing the retry interval. At first you will see ErrImagePull, but it may change. Rather than repeating the status names, read the Failed message in Events to see which point of address, authentication, or communication failed.
The target to check is the node where the Pod is deployed. Even if I can receive the image from my laptop, the DNS, proxy, and certificate conditions may be different on that node. Since events may disappear depending on the retention period, it is best to record the Pod name, UID, node, and time of failure when a failure occurs.
| clues to the event | Where to check first |
|---|---|
not found, manifest unknown | Existence of registry/project path/image tag |
unauthorized, pull access denied | Image path and account permission/authentication information. Also check for path errors in private repositories. |
FailedToRetrieveImagePullSecret | Secret name and Pod namespace |
no such host, i/o timeout | DNS, proxy, firewall, and registry status of the node |
x509: certificate signed by unknown authority | CA trusted by the node’s container runtime |
no matching manifest, HTTP 429 | Node architecture or registry request limits |
Image name and tags are read from the running Pod
kubectl get pod POD_NAME -n demo \
-o jsonpath='{range .spec.initContainers[*]}init/{.name}{": "}{.image}{"\n"}{end}{range .spec.containers[*]}{.name}{": "}{.image}{"\n"}{end}'We do not assume that the name written in the local YAML file is the same as the name of the actual Pod. You may have put other tags in your Helm values, Kustomize, GitOps repository, or deployment pipeline. Even if the image of the init container fails, the body container does not start, so check it together.
When changing the image, rather than editing the Pod directly, modify the template of the managed resource, such as the Deployment you own. Below is an example assuming that the approved image actually exists in container web in Deployment web. In production environments, validated tags or digests are used.
kubectl set image deployment/web \
web=registry.example.com/team/web:1.2.3 \
-n demoModify the image in the declaration’s source repository, if managed by a GitOps tool like Argo CD. This is because temporarily executed kubectl set image may be returned. There is no need to repeatedly delete multiple Pods just to fix a simple typo.
Authentication Secret must be in the same namespace as the Pod
kubectl get pod POD_NAME -n demo \
-o jsonpath='{.spec.imagePullSecrets[*].name}{"\n"}'
kubectl get secret regcred -n demo \
-o jsonpath='{.type}{"\n"}'Check that the name you registered for imagePullSecrets is correct and that the Secret is in the same namespace. regcred is an example name. Even if a Secret exists, you cannot receive it if you do not have account permissions or the token has expired. Even if the local docker login is successful, the authentication is not automatically passed to Kubernetes.
If you need a new Secret, you can prepare the Docker config file for your registry in a safe path and create it as shown below. The file must contain a valid certificate from the registry. Replace the example path with the actual file path, and Secret creation permission is required.
kubectl create secret generic regcred \
--namespace demo \
--type=kubernetes.io/dockerconfigjson \
--from-file=.dockerconfigjson=/secure/registry/config.jsonIf your local Docker config only points to an external credential helper, the authentication values may not be in the file. Just copying those files is not enough. Prepare authentication for actual deployment, and do not place the original file, Secret YAML, or token in the Git repository or public log. If a Secret with the same name already exists, this creation command will fail and will be processed through the organization’s renewal procedure.
Merge the items below into the Pod template of Deployment. It is not a full deployment document and does not erase and replace existing settings such as images or volumes. If the configuration is connected through ServiceAccount, check the items reflected in the actual Pod.
spec:
template:
spec:
imagePullSecrets:
- name: regcredCommunication issues are checked at the node receiving the image
Image downloading is performed by the node’s kubelet and container runtime. Accessing the registry from another running Pod does not determine that the node’s image download path is normal. The pod’s internal DNS and the DNS conditions used by the node may be different.
If administrative rights and operating procedures allow, check the problematic node as follows: Replace the address with your actual registry. The CA used by curl and the trust store of the container runtime may be different, so the results are read as a guide.
getent hosts registry.example.com
curl --connect-timeout 5 --max-time 10 \
-I https://registry.example.com/v2/
sudo journalctl -u kubelet --since '30 minutes ago' --no-pager
# containerd를 사용하는 노드인 경우
sudo journalctl -u containerd --since '30 minutes ago' --no-pagerAn HTTP 401 coming from /v2/ in a registry that requires authentication may be a clue that communication has arrived. That response alone does not confirm permission to read the image. Since there are services that download image layers from other domains, also check the actual destination shown in the event and runtime log.
For a private CA error, fix the certificate trust configuration used by the runtime. Do not make disabling TLS verification the default solution. A change that requires restarting the node runtime can affect other Pods; plan draining and maintenance accordingly.
Check deployment conditions before changing imagePullPolicy
Always does not resolve authentication or network errors. IfNotPresent can skip downloading when a node has the image it needs, but the new node still needs the image. Never may cause other failures on nodes without images.
If it is a closed network, first decide how the image will be prepared on all target nodes or the internal registry path. Downloading the image to Docker on the development PC and preparing it at Kubernetes node runtime are separate things. Avoid masking missing images and authentication issues by changing your download policy.
After modification, view the new Pod’s events and service responses
kubectl rollout status deployment/web -n demo --timeout=120s
kubectl get pods -n demo -o wide
kubectl get events -n demo --sort-by=.metadata.creationTimestampAfter confirming that a normal image has been received, check whether the container has started, is in a Ready state, and is responding to actual requests. After receiving the image, if it changes to CrashLoopBackOff, now examine the logs of the application startup phase and the cause of its termination. A full recovery is not recorded simply because the status has changed.
This article is a diagnostic procedure based on Kubernetes official documentation. This is not a result of reproducing a private registry/certificate failure in an actual cluster. Diagnosis after Pod execution can be seen in Pod failure diagnosis and node drain.
reference document
Kubernetes: Image and ImagePullBackOff
Kubernetes: Get private registry image
