Prefill and Decode: why asking an LLM for three takeaways from a long document still takes minutes
dotey · x · 2026-09-15
- This article uses the everyday scenario of feeding a long project doc to an AI minutes before a meeting—then waiting despite asking for just three takeaways—to explain LLM inference's two stages.
- Prefill must process the entire long input before generation begins, while Decode emits tokens one by one; the compute-heavy prefill over a long document, not the short requested output, dominates the wait.
More from Infra
- Leaker Claims Nvidia RTX Rubin 6090 Launching Next Year — max_paperclips · 2026-09-15
- Pareta routes cheap LLM tasks to small models, 620x cheaper than GPT-5.5 — D33B · 2026-09-15
- Training from scratch on a single H100 hits 76% on ARC-AGI-1 in ~4 hours — GregKamradt · 2026-09-15
- From Ollama to vLLM: a roadmap for scaling LLM deployment — kalyan_kpl · 2026-09-15
- Kimi K3 is live and free on NVIDIA NIM with OpenAI-compatible API — airesearch12 · 2026-09-15
- Dual Radeon AI Pro R9700 vs. Two Used RTX 3090s at $1600 Each for Local LLM Inference — Current-Ticket4214 · 2026-09-15