Understanding LLM inference: How Prefill and Decode phases impact speed
techNmak · x · 2026-08-12
This post breaks down the underlying mechanics of LLM inference latency by splitting the process into two core phases:
- Prefill Phase: The model tokenizes the input prompt and processes it through transformer layers with heavy parallelization, building the KV cache. The computational work here is the primary contributor to Time To First Token (TTFT). Longer prompts mean more prefill work before generation begins.
- Decode Phase: The model generates new tokens one by one. This is a strictly sequential process (token N depends on token N-1) that reuses and updates the KV cache. The speed of this phase is measured by Tokens Per Second (TPS).
The author notes that beyond input/output lengths, factors like model architecture, memory pressure, hardware bandwidth, and batch size significantly impact inference speed and cost. Thus, when an LLM feels slow, it's crucial to distinguish whether the bottleneck is waiting for the first token (Prefill) or slow subsequent generation (Decode).
Related event: Explaining LLM Inference: How Prefill and Decode Stages Affect Speed(2 posts)→
More from Infra
- AI-Driven Timing-Aware Power Optimization Uses Learned Ranking With Baseline-Preserving Repair — Amit Sharma · 2026-10-16
- Nscale AI datacenter plagued by coolant leaks, transformer fire, 10 safety violations — saranormous · 2026-10-16
- WSL 3.0 ships with native containers and Linux 6.18 kernel, syscall overhead down ~16% — solyarisoftware · 2026-10-16
- Are 22 GB modded RTX 2080 Ti cards a viable budget local LLM option? — Dave86ch · 2026-10-16
- Intel's neural-compressor: SOTA low-bit LLM quantization with FP4/NVFP4 support — HaihaoShen · 2026-10-16
- Tencent squeezes 770B-parameter Hunyuan Hy4 Preview into 214 GiB at ~2.38 bits per weight — coolmum18 · 2026-10-16