A write can fail with No space left on device even when the disk usage display is below 100%. Before deleting more data, identify the filesystem containing the failing path and check its inode availability. Capacity, quotas and files still held open require different investigations.
The commands below are based on Linux that uses GNU tools, such as Rocky Linux and Ubuntu. An example path is /var/log. If the application fails while writing a file, change to the actual storage directory of the program and run it.
On this page
Check df in the error path
df -h /var/log
df -i /var/log
findmnt -T /var/logdf -h shows the capacity of the file system to which the path belongs, and df -i shows the inode usage. The inode is a structure that contains information such as the file owner, permissions, and data location. Empty files also use inodes, so if there are many small files, file creation can be blocked independent of the remaining bytes.
Just by looking at the server-wide df -h output, it is easy to feel reassured by the free space on other disks. Even if there is space in /, /var, the write layer of the container, and tmpfs, which has a limited size, may become full. Look at findmnt, TARGET, SOURCE, and FSTYPE and guess what you are checking now.
| Verification result | Where to See Next |
|---|---|
There is almost no Avail in the target file system. | Check for large directories in the same file system and the cause of their growth |
IFree is missing or inode utilization is very high | Check directories containing cache, sessions, and small files |
| Deleting files does not reduce usage | Check which processes are still opening deleted files |
| There is room on the host, but only the container fails. | Check path/mount/tmpfs restrictions inside container |
The method of allocating inodes varies depending on the file system. Do not directly apply ext4’s inode exhaustion case to XFS, but judge it based on the file system in use and the actual output. Exceeding user quotas usually appears as a separate error called Disk quota exceeded, so keep them separate.
If capacity is insufficient, it is narrowed down within the same file system.
sudo du -x -h --max-depth=1 /var/log | sort -hdu counts the usage of files accessible under a directory. -x prevents it from moving to another file system. If you find a large subdirectory, simply repeat the same command targeting that path. This reduces the search scope and disk read burden compared to scanning the entire system each time.
If there is a read permission error, the aggregation may be missing, so look at the error as well. df and du do not have to be exactly the same. There are also implications for file system metadata, reserved space, snapshots, shared blocks, etc. Just because there is a big difference does not mean that the file system is damaged.
Even if a large file is found, first check whether the data is still being used by the program. Deleting database files or container storage directories directly with file management commands can make recovery difficult. Logs are rotation and storage settings for the relevant service, and container data is organized after checking the owned service and volume.
If you run out of inodes, look at the number of files rather than the large ones.
sudo du -x --inodes --max-depth=1 /var | sort -nHere, inode usage is given instead of size. Try going back down to the directory with the larger number. It can help you find where small files accumulate, such as session files, mail queues, and task temp files. In tool environments without du --inodes, first check for support with du --help.
Even though there are many files, not all of them are unnecessary. Check the creation program, storage period, current use, and use the organizing functions provided by the program. If the creation rate is faster than the file deletion rate, retry loops and expiration processing failures must also be fixed.
A process may still be holding onto deleted files
In Linux, deleting a file name may not immediately release the data if a process has the file open. This is a typical situation where du disappears when scanning a directory, but df usage remains. On the server where lsof is installed, check as follows.
sudo lsof +L1+L1 picks open files with link count less than 1. Look at COMMAND, PID, SIZE/OFF, and NAME in the output to find the process holding the large file on the filesystem in question. The absence of output does not rule out all causes of space problems.
Once the target service is identified, use the reopen log feature or normal restart procedure. You should check the service impact first, as a restart may lead to connection interruptions or operation failures. Writing directly to /proc/.../fd/... or forcefully killing the process is not the default cleanup procedure.
In containers, internal paths are also checked separately.
Verify that the host’s disk and the storage space seen by the container are in the same condition. Replace myapp below with the name of the container you want to investigate. If there is no shell or df in the container image, the host-side storage path is investigated based on the mount information of docker inspect.
docker inspect myapp --format '{{json .Mounts}}'
docker inspect myapp --format '{{json .HostConfig.Tmpfs}}'
docker exec myapp df -h /tmp
docker exec myapp df -i /tmpIf the location where the file is being written is /tmp, the limited memory file system, not the disk, may be full. If it is a bind mount, check Source of Mounts on the host. Match it with Destination in the container to avoid cleaning up other volumes with similar names.
After taking action, check the cause of file creation and increase.
After cleaning, check df -h and df -i again in the same path, and rerun the program operation that failed. Even if file creation returns to normal, the same problem will recur if logs or temporary files continue to grow rapidly. If you leave the storage period and the results of the cleaning work, the next inspection will be easier.
If the capacity is insufficient due to actual data growth, you can check the procedure suitable for the current structure in LVM·XFS·ext4 root extension or LVM extension partition.
While preparing this article, I confirmed that open file descriptors were maintained after deleting an 8 MiB file from a separate temporary directory, and that inode usage increased due to the creation of an empty file. We did not experiment with filling up the operating disk or deleting actual service files.
