Understanding LLM inference: How Prefill and Decode phases impact speed
techNmak · x · 2026-08-12
This post breaks down the underlying mechanics of LLM inference latency by splitting the process into two core phases:
- Prefill Phase: The model tokenizes the input prompt and processes it through transformer layers with heavy parallelization, building the KV cache. The computational work here is the primary contributor to Time To First Token (TTFT). Longer prompts mean more prefill work before generation begins.
- Decode Phase: The model generates new tokens one by one. This is a strictly sequential process (token N depends on token N-1) that reuses and updates the KV cache. The speed of this phase is measured by Tokens Per Second (TPS).
The author notes that beyond input/output lengths, factors like model architecture, memory pressure, hardware bandwidth, and batch size significantly impact inference speed and cost. Thus, when an LLM feels slow, it's crucial to distinguish whether the bottleneck is waiting for the first token (Prefill) or slow subsequent generation (Decode).
More from Infra
- Open Source AI Summit Announces Talk on LLM Inference Engine Optimization — zainhas · 2026-08-12
- Google's PROMPTS: Multi-Agent Framework Boosts LLM Training Performance by 434% — rohanpaul_ai · 2026-08-12
- Google Paper: LLM Infra Optimization via Agent-Driven Bottleneck Analysis — rohanpaul_ai · 2026-08-12
- Running MiniMax H3 on Low VRAM Fried My GPU, Beware — ROBOTTTTT13 · 2026-08-12
- StarCloud Explores Space Data Centers: Launching AI Hardware into Orbit — DavidLinthicum · 2026-08-12
- gakonst adds confidential compute support to nanocodex for verifiable confidential AI — AccBalanced · 2026-08-12