GGUF VRAM calculator: the three numbers that decide
published: true title: GGUF VRAM calculator: the three numbers that decide tags: discuss, ollama, gguf, vram canonical_url: https://mrsaynothing.dev/en/space/2026-10-31-gguf-vram-calculator-the-three-numbers-that-decide cover_image: https://mrsaynothing.dev/api/space/media/post/c1b06cac-ceac-4f7f-91fa-8622c66eb69e
Cross-posted from Mr Say Nothing — receipts live there.
"Will this quant fit?" is a three-number question, not a feeling: weights + KV cache + overhead. I printed the formulas and shipped them as a calculator you can open from disk — then checked the math against four public GGUF files. Worst miss: 0.04 GB, under one percent. Here is the whole decision on paper, so the allocator never gets to be the one who tells you.
The three lines
weights = params × 1e9 × (quant bpw ÷ 8) × 1.08
kv cache = 2 × layers × kv_heads × head_dim(128) × 2 bytes × context
overhead = CUDA context + cuBLAS + compute buffers (≈ 0.6–1.2 GiB)
bpw is bits per weight — what a quant spends on one parameter. The KV cache is the model's scratchpad for the conversation so far, and it grows with every token of context. Overhead is what the GPU keeps for itself.
The 8% on the weights line is not fudge for fudge's sake: embeddings and the output head stay above the quant's nominal bits-per-weight in real files, and the factor soaks that up. A formula you can't explain is a horoscope; this one fits inside a tweet.
Does the math survive contact with real files?
| Model · quant | Real file | This math | Drift | | --- | --- | --- | --- | | Llama-2 7B · Q4_K_M | 4.08 GB | 4.09 | +0.01 | | Llama-3 8B · Q4_K_M | 4.92 GB | 4.88 | −0.04 | | Mistral 7B · Q4_K_M | 4.37 GB | 4.39 | +0.02 | | Llama-3 8B · Q8_0 | 8.54 GB | 8.55 | +0.01 |
Max drift 0.04 GB, on files anyone can download and weigh. Three of the four sizes re-pulled live from the Hugging Face API on 2026-10-31.
One unit slip worth printing, because I made it myself: the tool's README called that column GiB, but model files list decimal GB — 4.92 GB is 4.58 GiB, and that 0.34 difference is a third of an 8 GB card. The card's monitor counts GiB; the file's size is quoted in GB. Now both do.
The KV cache is the number everyone forgets
Llama-3 8B: 32 layers, 8 KV heads, head_dim 128:
2 × 32 × 8 × 128 × 2 = 131,072 bytes/token = 128 KiB/token
- 8k context → 1.0 GiB
- 16k context → 2.0 GiB
- 32k context → 4.0 GiB
The file never grows. The bill does. That 4.9 GiB model on an 8 GB card is comfortable at 8k (~6.2–6.8 GiB total) and impossible at 32k (~9.5 GiB) — same weights, same card, four more GiB of scratchpad. Every "it fit yesterday and OOMs today" thread is this table wearing a trench coat.
Older 7B models are hungrier at context than newer ones: Llama-2 7B runs full multi-head attention — 32 KV heads, not 8 — so the per-token bill is 4× bigger, 512 KiB/token, a full GiB per 2k tokens. When an old guide's context math feels insane, check the KV-head count before blaming your GPU.
If you load through text-generation-webui
The knobs there are exactly the three numbers: n_ctx (feeds the KV cache line), n-gpu-layers (how much of the model lives on the card at all), and cache_type — dropping the KV cache from fp16 to q8_0 halves that line at a small quality cost. One dry truth about loaders in general: they will happily begin loading a file the card cannot finish holding. The calculator exists so the allocator never gets to be the one who tells you.
What the math won't cover
Three honest blind spots, printed on the tool itself: MoE routing (mixture-of-experts models shuffle experts in ways per-layer math doesn't see), draft models (speculative decoding parks a second model in the same memory), and your desktop's share — your tabs are not in the formula, but they are in your task manager.
The calculator is one static page — no build, no network, open it from disk: github.com/mrsaynothing/gguf-vram-calculator.
The open wound
MoE sharding is where this math goes blind, and mixture-of-experts is exactly where the local-LLM world is heading. Until the formula learns to see experts, the 0.04 GB promise holds only for dense models — and the honest thing is to say so.
So, those of you running MoE quants locally: how far off is this arithmetic on your machine, and what would the experts line even need to count?