Building a Local LLM · 05/30 · Basics
Full series and learning sequence · Practice code ZIP
When a long conversation loses earlier information, inspect the context used by the runtime. The input may have been truncated by a small limit, or the model may have overlooked relevant information in a long input. Tokens and context length help explain these cases alongside their memory requirements.
Leave Ollama running and open PowerShell. Decide how many tokens to use for input and output, and see how memory changes when you increase the context.
Tokens and Input Limits
The model processes numerical units, or tokens, divided by a tokenizer. One Korean letter or one English word is not always one token. Even for the same sentence, the number of tokens varies depending on the model. When calculating input limits, we consider not only the document but also system instructions, previous conversations, tool definitions, and output space.
If the context is 8192 and 8000 tokens are used for input, there is no room to store 2000 tokens of answers. Depending on the app, the request may be rejected, the previous content may be omitted, and the output may be shortened. Before deciding on the length of the material to be included, leave space for answers.
API requests specifying an execution context
$body = @{
model = "qwen3:4b-instruct"
messages = @(@{role="user";content="백업과 스냅샷의 차이를 두 문장으로 설명해 줘."})
stream = $false
options = @{num_ctx=4096;num_predict=160;temperature=0.2}
} | ConvertTo-Json -Depth 6
$r = Invoke-RestMethod -Uri http://127.0.0.1:11434/api/chat -Method Post -ContentType 'application/json; charset=utf-8' -Body ([System.Text.Encoding]::UTF8.GetBytes($body))
$r.message.content
$r | Select-Object prompt_eval_count, eval_countYou can see the number of input tokens actually processed with prompt_eval_count. Check the maximum context of the model card and the execution context specified by num_ctx separately. Even models that support large contexts are difficult to run at that size if the device has insufficient memory.
KV cache and memory
Transformer’s attention calculates which part of the previous token the current token should refer to. At this time, the KV cache is a storage space that stores the key and value of the previous token to avoid repeating calculations. In general, as the number of contexts and concurrent sequences increases, the cache also grows. GQA·MQA is a structure that lowers this burden by reducing the number of KV heads to be stored.
The size of the cache depends on the model structure and cache precision. Rather than memorizing one number for memory per token, try changing the context from 4K to 8K on the same model. Recording memory and first response time together makes it easier to determine the range your PC can handle.
Differences in capacity and quality of quantization
Quantization is a method of approximating FP16 weights to fewer bits to reduce storage and bandwidth burden. As the value changes, there may be a difference in the answer. Q4_K_M also does not mean that all numbers are equally cut into 4 bits. This is a notation that indicates both the quantization method and the processing method of important tensors.
ollama show qwen3:4b-instruct
ollama show qwen3:4b-instruct --modelfileDifferences are confirmed using the same evaluation questions. Compare whether you got the answer right and followed the requested format. Depending on the task, a small, high-precision model may be better than a large, low-precision model. It is easy to interpret the results if you do not change size, precision, and context all at once.
reference material
Documentation and model tags checked: October 18, 2026. Primary lab environment: Windows 11 x64.
