[Local LLM 26] Measure Loading, First Response and Generation Speed

Building a Local LLM · 26/30 · Advanced and Operations

Full series and learning sequence · Practice code ZIP

Separate the time spent loading model files, processing input and generating output tokens. A slow result in each stage can have a different cause. Repeat the same question while recording each interval, keeping initial loading separate from a request to an already loaded model.

Use the same practice model as the function in Part 13. We compare model loading, input processing, and output generation separately.

Record measurement conditions

Fix model tags and IDs, quantization, context 4096, and output limits. Also take note of power modes and other GPU activity. Repeated requests for the same question may be affected by the prompt cache. The laptop also includes whether it is on battery or connected to power.

ollama ps
ollama stop qwen3:4b-instruct

The first request after stop may take some time to load the model. The later request may have the model remaining in memory. It is easier to explain the difference in experience if you record it separately when you run it for the first time and when you use it continuously.

Measuring first content and creation speed

import json
import time
import urllib.request
from llm_http import BASE, MODEL

payload = {"model": MODEL,
    "messages": [{"role": "user", "content": "백업 보관 정책에서 확인할 항목을 5가지로 설명해 줘."}],
    "stream": True,
    "options": {"num_ctx": 4096, "num_predict": 192, "temperature": 0}}
for run in range(1, 4):
    req = urllib.request.Request(BASE + "/api/chat",
        data=json.dumps(payload, ensure_ascii=False).encode("utf-8"),
        headers={"Content-Type": "application/json"})
    start = time.perf_counter()
    first = None
    final = None
    with urllib.request.urlopen(req, timeout=300) as response:
        for line in response:
            if not line.strip():
                continue
            item = json.loads(line)
            if item.get("message", {}).get("content") and first is None:
                first = time.perf_counter() - start
            if item.get("done"):
                final = item
    elapsed = time.perf_counter() - start
    if not final:
        raise RuntimeError("Missing final stream message")
    seconds = final.get("eval_duration", 0) / 1e9
    print(json.dumps({"run": run, "first_content_seconds": first,
        "wall_seconds": round(elapsed, 3),
        "load_seconds": final.get("load_duration", 0) / 1e9,
        "prompt_tokens": final.get("prompt_eval_count"),
        "output_tokens": final.get("eval_count"),
        "decode_tokens_per_second": final.get("eval_count", 0) / seconds if seconds else None},
        ensure_ascii=False))

.\.venv\Scripts\python.exe .\benchmark.py

first_content_seconds is the time the client received its first content fragment. It is different from the header arrival time or the first token calculation time inside the server. decode_tokens_per_second is the number of generation tokens reported by the server divided by the generation time, divided by the total request throughput.

Compare settings one by one

symptoms	Experiment to do first
Only the first request is slow	Comparison of load_duration and model residence time
First output is delayed for long documents	Compare prefill by halving the input length
Slow throughout output	Check GPU load ratio and memory bandwidth
Entire PC stutters	Check RAM pressure, page file, and concurrent apps
Worse in multiple users	Comparison limited to one concurrent request

We do not change context, model size, quantization, or concurrency all at once. Try evaluating Part 18 again with increased speed settings. If number or formatting errors increase, it is difficult to say that it has become faster while maintaining quality.

Check logs by symptom

Windows log location follows the %LOCALAPPDATA%\Ollama instructions in the official documentation. Connection errors are checked starting from server execution, and model errors are checked from tags and downloads. Memory errors look at context and resident models, as well as GPU support. We do not disclose the log keys or the original text of the conversation.

Keep the three lines of results and execution conditions from the example together. If you can repeat measurements, compare the median value and the slowest section. A quick first response on a private screen versus handling many requests on a shared server may have different goals.

reference material

• Ollama creation time field
• Ollama Streaming
• Ollama Troubleshooting

Documentation and model tags checked: October 18, 2026. Primary lab environment: Windows 11 x64.

---