[Local LLM 27] Compare GGUF and CPU/GPU Options in llama.cpp

Building a Local LLM · 27/30 · Advanced and Operations

Full series and learning sequence · Practice code ZIP

llama.cpp lets you select a GGUF file and inference options directly. Run the model on CPU, request GPU offloading and compare the logs. This makes visible some of the model and runtime settings previously managed by Ollama.

Prepare space to store Windows prebuilds and models. If 8B in the example is difficult to implement, choose the smaller supported GGUF model.

GGUF and supporting engines

GGUF contains weights and associated metadata. Even if the file format is the same, the engine must support the model structure and tokenizer to run it. Also check the quantization format and dialogue template.

Get the right Windows build for your device and drivers from Official Releases. Check the differences in builds such as CPU, CUDA, Vulkan, etc. and then extract them. Run the command below from the folder where llama-cli.exe and llama-server.exe are located.

CPU execution options

.\llama-cli.exe --help
.\llama-cli.exe -hf Qwen/Qwen3-8B-GGUF:Q4_K_M -c 4096 -n 128 -ngl 0 -p "Explain why backups need a restore test. /no_think"

-hf is an option to receive and use a quantization file from the repository. Internet and storage space are required for the first run. -c is the context, -n is the generation limit, and -ngl 0 is a comparison condition that leaves the GPU loading layer at 0. If you feel uncomfortable with a large model, check with a small file first.

This model has different weights from the 4B Instruct in Part 2. Don’t interpret the difference in answers as just an engine difference. Leave the download path, file hash, and binary version so you can recreate the same conditions.

GPU loading and startup logs

.\llama-cli.exe -hf Qwen/Qwen3-8B-GGUF:Q4_K_M -c 4096 -n 128 -ngl 99 -p "Explain why backups need a restore test. /no_think"

-ngl 99 is the value in this example that requests GPU loading of many layers. The actual number of loads and devices used can be checked in the startup log. Increasing VRAM is not an option, so you’ll also need to look at whether your GPU is built and the low memory message.

Compares with CPU path with same file/context/input, leaving total time and number of output tokens. The speed of prompt processing may differ from the speed of sequential output.

local API server

.\llama-server.exe -hf Qwen/Qwen3-8B-GGUF:Q4_K_M --host 127.0.0.1 --port 8081 -c 4096 -ngl 0

In another PowerShell, check the model ID with Invoke-RestMethod http://127.0.0.1:8081/v1/models. You can exit the server using Ctrl+C in the window where it is running. When sending a request, we use this server’s API format and do not send Ollama’s /api/chat code as is.

If you run the engine yourself, you are also responsible for model management, server startup, logs, and updates. It is convenient for comparing settings in detail, but the convenience features required for personal chat may be different. Choose a runtime based on what tasks you want to repeat.

reference material

• llama.cpp Official Project
• official distribution file
• Qwen Official GGUF and Implementation Guide

Documentation and model tags checked: October 18, 2026. Primary lab environment: Windows 11 x64.

---