MaruOpsInfrastructure and AI notes

About (Korean)Contact (Korean)

Diagnose a Restarting Docker Container with Logs and Exit Codes

Server cabinet of the TSUBAME 4.0 system

Reference photo · Server cabinet of the TSUBAME 4.0 system · Photo: Fukumoto / Source / CC BY-SA 4.0 · Resized for listings; illustrative photo, not the tested environment

When docker ps repeatedly shows a container as Restarting, capture its recent logs and exit state. A failing start command, missing file and memory shortage call for different corrections. A restart policy starts the process again but does not fix the condition that made it exit.

Below is based on Docker Engine and Docker Compose v2 on Linux. myapp is an example of a container name, and app is an example of a Compose service name. Since the two names are not always the same, check at docker compose ps -a first.

On this page

Leave the status and log at the time of failure first.

docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}'
docker logs --timestamps --tail 100 myapp
docker inspect myapp --format '{{json .State}}'
docker inspect myapp --format 'restarts={{.RestartCount}} policy={{.HostConfig.RestartPolicy.Name}}'

View State together with ExitCode, OOMKilled, Error, StartedAt, FinishedAt. The moment the container runs again, its current state changes, so it must be checked against logs from the same time. Also check if RestartCount continues to increase.

Even if the log is empty, it doesn’t mean it’s normal. The program may write to an internal file instead of standard output or standard error, or may fail before starting the executable. At this time, check State.Error and Docker daemon log. When sharing research data, secret values ​​such as log tokens and connection strings must be obscured.

ClueInterpretation and next confirmation
ExitCode=0, but repeated executionCheck whether the main process is a one-off task that terminates normally or has the always policy.
Non-zero exit code and setup error logCheck required settings, environment variables, and execution commands. Code semantics contrasted with program documentation
OOMKilled=trueCheck container memory limits and host memory status
Exit code 137Clues related to SIGKILL. Additional confirmation of OOM through status and kernel logs
unhealthy but still runningInvestigating health screening failures. The normal Docker restart policy does not restart in this state alone.

When to stop the restart loop

If requests are made repeatedly to an external system or logs accumulate quickly, only the container under investigation is stopped after the impact is checked. The following command will stop servicing that container: There is no need to use docker compose down as the first step in diagnosis, as it drops other services as well.

docker stop myapp

When stopped manually, the normal Docker restart policy does not restart the container immediately. However, the always policy has different behavior when the Docker daemon is restarted. If the container is managed by Kubernetes, Swarm, or an external monitoring program, it must be stopped or changed by the management entity.

Match execution commands and mounts before environment variables.

docker inspect myapp --format 'entrypoint={{json .Config.Entrypoint}} cmd={{json .Config.Cmd}}'
docker inspect myapp --format '{{json .Mounts}}'
docker compose config --quiet

Make sure you overwrite the image’s default command with Compose’s command. If the executable file path, arguments, or working directory are different, startup may fail even if the installation is normal. In environments where secret values ​​are entered as command arguments, they may also be included in the output above, so they are not pasted as is.

The mount checks Source, Destination, and RW. Check if an empty directory on the host covers a configuration directory in the image, or if a path that requires writing is connected as read-only. It is best to check environment variables against Compose files and program documentation, but not output the entire settings to a public log.

If you see Permission denied, match the ownership of the host directory with the user running the container. Rocky Linux also checks SELinux’s denial history. If you apply chmod -R 777 or turn off SELinux without checking the cause, it is easy to hide the problem location. For volumes with existing data, check the scope of impact before making wholesale changes to ownership.

Distinguish between memory termination and dependent service failure

docker inspect myapp --format 'oom={{.State.OOMKilled}} memory_limit={{.HostConfig.Memory}}'
docker stats --no-stream myapp
free -h
sudo journalctl -k --since '30 minutes ago'

HostConfig.Memory is in bytes, and 0 means that no separate memory upper limit is specified for the container. This does not mean that there are no host or parent cgroup restrictions. docker stats is a value at the moment it is running, so it does not show the peak usage immediately before it has already ended.

If the log indicates a database connection failure, check the address, port, authentication, and preparation status separately. Compose’s simple depends_on does not assume that the database is ready to receive requests. The health check conditions and the application’s retry behavior must be designed together.

If you changed the Compose settings, create the container again.

After modifying the environment variables or commands, simply run docker restart to restart with the existing container settings. In the project directory that manages Compose files, update the service as shown below. Service connectivity may be lost during regeneration.

docker compose config --quiet
docker compose up -d --no-deps app
docker compose ps -a
docker compose logs --since 5m --tail 100 app

--no-deps is an option to not start dependent services together. Used when the dependent service is already operating normally and its settings have not been changed. If the image or executable file itself has been modified, a separate image building and distribution process is also required. down -v or brute force cleaning of data volumes is not required for this diagnostic process.

Up sign is followed by confirmation of the actual request

Verify that the container shows Up and does not increase the number of restarts. If you have a health test, see the results and send an actual request to the application as well. You must separately check that the container’s process is alive and that the service is responding normally.

We intentionally configured a separate Python container to return exit code 42 to check the number of restarts and look up the log. This experiment verifies how to read state and does not replicate specific database failures or memory exhaustion. If you need host service logs, you can also see systemd and journalctl usage instructions.

reference document

Docker: Container Restart Policy

Docker logs

Docker inspect

Docker: Memory resource limitation

Compose: Service startup sequence

docker compose up