If a model answers while the CPU is busy and GPU utilization is low, check ollama ps before reinstalling drivers. A missing GPU and partial CPU offloading require different investigations. A utilization reading taken after a short request can also be misleading.
These steps apply to Linux with an NVIDIA GPU. AMD ROCm and Apple Metal have different tools and support requirements. Check host device visibility → Ollama model placement → server environment in that order.
On this page
Confirm that the host can see the GPU
nvidia-smi
nvidia-smi -L
ollama --versionIf nvidia-smi fails or cannot communicate with the driver, inspect the host driver, kernel modules, and device assignment. In a virtual machine, verify GPU passthrough to the guest. Changing model settings will not resolve a host-level detection failure.
When the GPU appears, record its name, driver version, total VRAM, and other processes’ memory use. The CUDA version at the top of nvidia-smi indicates the version supported by the driver; it does not establish that the same CUDA Toolkit version is installed on the host.
Compare the GPU and driver with the current Ollama hardware support table. Requirements can change between releases, so no single driver version is presented here as a universal fix.
Read ollama ps while the model is loaded
Find an installed model with ollama list. Replace MODEL_NAME below with its exact name and tag. Run it in one terminal, then inspect its state in another while the model is still loaded.
ollama list
# MODEL_NAME을 설치된 모델 이름으로 바꿉니다.
ollama run MODEL_NAMEollama ps
nvidia-smi| Observed state | Interpretation and next check |
|---|---|
No model in ollama ps | Confirm the target server and whether loading completed or the model was already unloaded. |
PROCESSOR shows 100% GPU | GPU placement is active. Investigate request length, loading time, and concurrent work separately if throughput is low. |
100% CPU | Check supported hardware, drivers, server environment variables, and device access. |
| Both CPU and GPU are shown | Check available memory, model size, and context settings. |
PROCESSOR describes how the loaded model is split between CPU and GPU. It is not an instantaneous GPU utilization percentage. Partial CPU placement alone does not prove that GPU detection failed.
If the CLI’s OLLAMA_HOST points to a remote server, ollama ps reports that server while local nvidia-smi reports your own computer. A host CLI and a containerized server can likewise target different environments. Confirm that both observations describe the same server.
Inspect server logs for discovery and memory loading
For a systemd installation, use the commands below. If you started ollama serve manually, read that terminal’s logs; for a container, use docker logs. Select the log source matching the installation.
systemctl status ollama --no-pager
sudo journalctl -u ollama --since '30 minutes ago' --no-pagerAround the failed request time, look for GPU discovery failures, driver initialization errors, available memory, and model loading messages. Wording varies by version, so the absence of one specific line does not establish CPU-only operation. Compare the logs with ollama ps.
Keep the GPU model, driver and Ollama versions, model name and quantization, context length, and logs from the same period together. A different model or another GPU job can change the result on the same server.
Check the service environment, not just the shell
systemctl cat ollama
systemctl show ollama --property=Environment --no-pagerAn environment variable exported in a terminal is not automatically applied to an already running systemd service. Inspect the drop-in configuration for an incorrect CUDA_VISIBLE_DEVICES value or a backend override left over from testing. For example, CUDA_VISIBLE_DEVICES=-1 prevents GPU use.
Environment and unit output can contain secrets such as proxy credentials; redact them before sharing. Correct only the confirmed problematic setting in its drop-in file. Removing every override can also remove model storage and server address settings.
If a change is needed, edit it with sudo systemctl edit ollama, then run the following commands. A restart can interrupt active requests, so check current work first.
sudo systemctl daemon-reload
sudo systemctl restart ollama
sudo journalctl -u ollama -n 80 --no-pagerPass the GPU through to the Docker container
Successful nvidia-smi output on the host does not prove container access. Check NVIDIA Container Toolkit and Docker runtime configuration, then inspect whether the Ollama container requested GPU devices. Replace ollama below with its actual container name.
docker inspect ollama --format '{{json .HostConfig.DeviceRequests}}'
docker logs --since 30m --tail 150 ollama
docker exec ollama ollama psIf DeviceRequests is empty, check whether devices were requested through the usual --gpus mechanism. Other device-passing methods exist, so this field alone does not describe every deployment. Compare the deployment with the official Docker GPU examples.
Restarting an existing container does not add a missing GPU request. Confirm that model data volumes will be retained before recreating it with updated settings. Begin with the identified configuration issue rather than changing Docker’s entire cgroup setup or reinstalling the host driver.
Compare memory and request settings
When the GPU is detected but some layers remain on CPU, inspect other VRAM consumers, model size, and context settings. Compare the same request with an already installed smaller model or a shorter context. Memory is needed for weights, context, and execution.
Coordinate with the owners of other GPU jobs rather than terminating them, and record one change at a time. The tokens, context, quantization, and memory guide (Korean) explains why a model’s file size is not its complete VRAM requirement.
If the issue occurs only after suspend and resume, retain the driver logs from that period. Module reload procedures in the official documentation can affect other GPU work and require a suitable maintenance window. This is a documentation-based diagnostic guide; it does not report a reproduced GPU failure or a hardware benchmark.
References
Ollama: hardware support and GPU selection
Ollama: troubleshooting and logs
Ollama: FAQ and model placement
Ollama: Linux service configuration
