How LLM inference actually works: from 140GB weights to GPUs and model serving, explained as a story
_jaydeepkarale · x · 2026-11-04
A storytelling-style primer on LLM inference infrastructure, walking through the full chain from model file to serving:
- A model is just a file of weights: a 70B model is 140GB — load it, run the math, get text back.
- CPU fails: inference needs billions of tiny multiplications at once; serial execution is far too slow.
- GPU bottleneck and fix: thousands of parallel cores, but system RAM is too far and slow, so cores idle — hence VRAM next to the cores, fast but small; a 16GB card can't hold a 140GB model.
- Hence model serving: distributing the model across cards/machines and serving it is where AI infra begins.
A solid intro for engineers without an infra background on why running large models requires GPU clusters and serving stacks.
More from Infra
- SpecPilot: Berkeley and NVIDIA's Agent Loop Auto-Optimizes Speculative Decoding per Deployment — UCBerkeley · 2026-11-04
- AMD exec joins Fireworks AI Forge event to highlight multigenerational partnership — AnushElangovan · 2026-11-04
- Microsoft details Outlook search overhaul: 7 relevance-ranked results and offline resilience — benjamingauthey · 2026-11-04
- AMD exec gifts DHH a 192GB AMD workstation, embraces Omarchy Linux — AnushElangovan · 2026-11-04
- Running 4x RTX 3090 on a consumer board: a Qwen3.8 local inference build — Civil_Fee_7862 · 2026-11-04
- LlamaIndex speeds up doc-parsing model OCRia-1 10x to 2800 tok/s on one B200 — llama_index · 2026-11-04