Building a Local LLM · 04/30 · Introduction
Full series and learning sequence · Practice code ZIP
For a local LLM, first check whether the model fits in memory. A fast GPU with insufficient VRAM may need to offload some weights to system RAM. Gaming performance figures alone do not describe the bottleneck that this arrangement introduces.
Prepare Task Manager and the model you received in Part 2. Let’s distinguish between weights, KV cache, and memory to reserve for execution.
Weight memory calculation
The starting point for estimating the weight size is number of parameters × number of bits ÷ 8. If the weight of the 8B model is stored as 4 bits, it will be about 4 billion bytes, or about 3.73 GiB. Here, block-specific scale of quantization and mixed precision information are added. The calculation should not be used as the minimum VRAM required.
| Additional Space Required | why you need it |
|---|---|
| KV Cache | Reuse the attention calculation results of already read tokens |
| execution buffer | Matrix operations and temporary tensor processing |
| Graphic screen/other apps | Windows and browsers also use GPU memory |
| concurrent requests | Increased cache when keeping multiple contexts together |
$parameters = 8e9
$bits = 4
[math]::Round(($parameters * $bits / 8) / 1GB, 2)Here, 1 GB in PowerShell means 2 to the power of 30 bytes. If you mix GB and GiB in the product specifications, the numbers for the same capacity will look different. When choosing equipment, check the units and add some margin to the actual runtime load.
CPU/GPU and memory bandwidth
CPU inference has the advantage of handling large model files because it can use system RAM. Instead, memory bandwidth can become a bottleneck during the decode step, which creates tokens one by one. GPUs take advantage of parallel computing and dedicated memory bandwidth, but you must first ensure that they fit within VRAM capacity. Increasing the number of CPU cores does not necessarily make the response proportionally faster.
We will look at prefill, which reads long input, and decode, which continues to write answers. The causes may be different when the first letter appears late and when tokens continue slowly after printing begins. How to measure the two sections separately is covered in Part 26.
GPU support conditions for each device
ollama run qwen3:4b-instruct
ollama ps
# NVIDIA 드라이버가 설치된 장비에서만 실행
nvidia-smiOllama’s GPU support table confirms exact product, operating system, and driver requirements. Even within the same NVIDIA/AMD product family, support conditions are different and the tables for Windows and Linux may not be the same. “16GB of shared GPU memory” on an integrated GPU is not the same resource as 16GB of dedicated VRAM.
Check before upgrading
I’m currently trying to run a small model with context 4096 on my PC. Take note of whether you are running out of RAM, running the entire model on the GPU, or the quality of your answers is poor. Increasing RAM can reduce memory pressure, but it will not make GPU operations themselves faster.
Partial offloading is a method of putting some layers on the GPU and others on the CPU side. You can run models that could not be run due to capacity, but there is also the cost of going back and forth between the two devices. If you are considering purchasing, it is helpful to first write down the target model, context length, and number of people who will use it simultaneously.
reference material
Documentation and model tags checked: October 18, 2026. Primary lab environment: Windows 11 x64.
