Where does VRAM go during LLM inference? Four buckets explained
blaizedsouza · x · 2026-09-18
akshaypachaar explains the four ways GPU memory is consumed during LLM inference — loading the model is only the first part.
- Model weights: fixed once loaded; precision (FP16 → INT8/INT4) is the biggest lever.
- KV cache: grows with context length and concurrent requests, since attention reuses stored key/value tensors.
- Two more buckets grow with batch size and concurrency.
Linked article: "How a GPU Actually Works", making quantization, speculative decoding, and continuous batching intuitive.
More from Infra
- GitHub rewrote the Copilot runtime in 800,000 lines of Rust, mostly by AI agents — DanWahlin · 2026-09-18
- Third-party audit reproduces Gensyn open-1b training step bit-for-bit — benfielding · 2026-09-18
- Anthropic Open-Sources Claude-Written GPU Optimizations Speeding 30+ Biomolecular Models ~4x — ResultBackground2450 · 2026-09-18
- Spotify: 777M users, 11-12M requests/sec — how AI changed its quality playbook — rseroter · 2026-09-18
- Anthropic open-sources 36 drop-in inference optimization kits for open bio-ML tools — AnthropicAI · 2026-09-18
- 605 new Linux kernel CVEs disclosed in one day, on top of 276 the day before — jedisct1 · 2026-09-18