Model weights are only 1 of 4 VRAM allocations: why loaded LLMs still OOM
blaizedsouza · x · 2026-10-13
A repost of @avichawla's article "How LLM Inference Works, Clearly Explained", addressing a common misconception: a model can load successfully yet run out of GPU memory on the first requests, because weights are only one of four VRAM allocations.
Key points:
- Model weights: loaded before inference, mostly constant. A 70B model needs 140 GB in FP16/BF16, 70 GB in INT8, 35 GB in INT4. Quantization helps, but kernels may still use higher-precision values for some operations.
- KV cache: every Transformer layer generates KV tensors per token so new tokens can attend to earlier ones without recomputation, growing with context length.
The article also breaks each generate() call into two phases: prefill (processing the prompt) is compute-bound, while decode (one token at a time) is memory-bound.
More from Infra
- Basalt: A fast Blackwell inference engine for Qwen 3.8 Flash Next with dynamic vision — jesdga95 · 2026-10-13
- 80B MoE Model Runs in a Chrome Tab: llama.cpp WebGPU Backed Pages Experts From Disk — naklitechie · 2026-10-13
- The End of the Best-Model-Wins Era: AI Value Is Rotating Into Data Centers and Apps — RihardJarc · 2026-10-13
- Market rotating out of semis into hyperscalers and application layer, analyst says trend will continue — RihardJarc · 2026-10-13
- Intel hikes CPU prices 15-20% across Panther Lake and older lines; AMD expected to follow — TheZachMueller · 2026-10-13
- Mistral models head to orbit in 50-satellite AI constellation with NVIDIA compute — MistralAI · 2026-10-13