Inference engineering is AI's most underrated skill: prefill vs decode explained
techNmak · x · 2026-10-28
Core argument
Training gets the attention, but serving real traffic surfaces a different class of problems: queueing, batching, KV-cache pressure, GPU memory, scheduling, and tail latency. Inference engineering is widely underestimated.
Inference is not one workload
- Prefill: processes the prompt and builds KV cache; parallelizes efficiently across positions, higher arithmetic intensity, compute-bound (pressure shows up in TTFT).
- Decode: advances autoregressively, repeatedly reading weights and KV state; at practical batch sizes it's memory-bandwidth-bound (pressure shows up in ITL/TPOT).
Where each optimization fits
- FlashAttention cuts memory traffic in attention; PagedAttention manages KV memory in blocks to reduce fragmentation
- MQA/GQA shrink KV heads, cutting cache storage and bandwidth
- Continuous batching lets requests join/leave the batch; chunked prefill breaks long prompts so they run alongside decode
- Prefix caching reuses KV for repeated prefixes (saves prefill compute, doesn't speed later decode)
- Quantization reduces weight and KV-cache memory/bandwidth; speculative decoding drafts with a cheaper model and verifies with the target—speedup depends on draft cost, acceptance rate, verification efficiency
Scale and metrics
Tensor parallelism adds communication, pipeline parallelism adds scheduling and bubbles, MoE adds routing and load-balancing; some systems disaggregate prefill/decode onto separate GPU pools, paying KV-transfer costs. Tokens per second isn't enough: production cares about TTFT, ITL/TPOT, throughput, queueing, concurrency, KV occupancy, GPU utilization, tail latency, and goodput—great aggregate throughput can still mean users waiting too long for the first token.
Takeaway
Inference engineering spans ML systems, distributed systems, GPU architecture, networking, and performance engineering: the model decides what token comes next; inference engineering determines what serving that token costs and how many users the hardware can support.
More from Infra
- RTX 5090M runs Gemma 4 31B at up to 4,154 tok/s prompt speed, iNordic shares configs — leonbollerup · 2026-10-29
- Microsoft Q3 blows past estimates: $92.3B revenue, Azure +48% YoY — RihardJarc · 2026-10-29
- Vercel launches CI that skips irrelevant tasks automatically, 2-6x faster — cramforce · 2026-10-29
- Google Cloud revenue soars 103% as model APIs hit 32B tokens per minute — rseroter · 2026-10-29
- Local AI video gen on an 8GB RTX 4060: a 14s scene rendered in ~31 minutes — cocktailpeanut · 2026-10-29
- SK Hynix surges 13% after doubling Q3 dividend to 20,000 won on Kioxia gains — TiernanRayTech · 2026-10-29