AILocal LLMGetting Started and Model Selection

How to Estimate GPU VRAM for Local LLM Inference

MaruOps

2026.10.23

Reference photo · A motherboard and CAMM2 memory hardware on display · Photo: 4300streetcar / Source / CC BY 4.0 · Resized for listings; illustrative photo, not the tested environment

Estimate local LLM GPU memory in three parts: model weights, the attention key-value cache, and runtime overhead. The downloaded model file is a useful starting point, but it is not a complete VRAM requirement.

The arithmetic below is for inference, not training. It gives planning estimates for a conventional decoder-only transformer; the actual requirement depends on the model architecture, quantization format, backend, context length, and concurrent requests.

On this page

Calculate a lower bound for the weights

weight bytes = parameter count × bits per weight ÷ 8
weight GiB = weight bytes ÷ 1,073,741,824

For a hypothetical model with exactly 8 billion parameters, the idealized weight storage is:

Weight precision	Decimal storage	Binary storage
16-bit	16 GB	14.90 GiB
8-bit	8 GB	7.45 GiB
4-bit	4 GB	3.73 GiB

These are calculated values, not measured model sizes. Real quantized formats also store scales and other metadata, and some tensors can retain higher precision. A model marketed as “8B” may not have exactly 8 billion parameters. Inspect the actual model files and configuration.

Quantizing the weights also does not automatically quantize the KV cache. Hugging Face documents weight quantization and cache strategies as separate choices.

Budget for context and concurrent sequences

The KV cache retains attention keys and values so the model can reuse earlier token information. For a standard full-attention model with uniform layers, a simplified estimate is:

KV bytes ≈ 2 × layers × KV heads × head dimension
           × cached tokens per sequence × concurrent sequences
           × bytes per cache element

The factor of two accounts for keys and values. Use the number of KV heads, which can be smaller than the attention-head count in grouped-query attention. For a hypothetical configuration with 32 layers, 8 KV heads, head dimension 128, one 8,192-token sequence, and 2-byte cache elements:

2 × 32 × 8 × 128 × 8192 × 1 × 2
= 1,073,741,824 bytes
= 1 GiB

This example is a calculation, not a benchmark. Sliding-window attention, hybrid architectures, cache quantization, prefix sharing, and backend allocation policies can change the result. A backend may reserve cache capacity beyond the tokens currently used. Budget for the total prompt plus generated tokens and the concurrency your service actually allows.

Leave room for the runtime and other GPU users

Activations, temporary buffers, kernels, and backend allocations also consume memory. A display server or another inference process can reduce the available VRAM before the model starts. There is no single overhead percentage that is reliable for every backend.

The Accelerate model memory estimator can help estimate model storage, but treat that as one input to your inference budget. It does not replace a test with the intended context and workload.

Measure the intended workload

On an NVIDIA system, record a baseline before loading the model, then observe it during loading and a representative request. This command samples device memory once per second:

nvidia-smi --query-gpu=name,memory.total,memory.used,memory.free --format=csv -l 1

For Ollama, check ollama ps while the model is loaded. It reports the CPU/GPU placement of the loaded model. Read it alongside memory observations and server logs; the placement percentage is not live GPU utilization.

ollama ps

Record the exact model tag and quantization, runtime version, context setting, number of simultaneous requests, and observed memory use. Test the longest request you intend to support. A successful short prompt does not establish that a larger context will fit.

Adjust the part that consumes the memory

If weights dominate, try a smaller model or a supported lower-bit quantization. If longer contexts trigger allocation failures, reduce the configured context or concurrency and compare again. CPU offloading and cache quantization are backend-specific options with performance and quality tradeoffs.

References checked on October 23, 2026. The numerical examples were calculated from the stated assumptions. No GPU benchmark or claim about a particular card fitting a particular model is made here.