Inference engineering is the underrated AI skill: KV cache, batching and p99 latency explained

techNmak · x · 2026-09-07

The author argues inference engineering is becoming one of the most important yet underestimated skill sets in AI. Training gets the spotlight, but serving real traffic surfaces a different class of problems: queueing, batching, KV-cache pressure, scheduling, kernel efficiency and tail latency.

Key points:

Prefill vs decode are fundamentally different workloads: prefill parallelizes well and is compute-heavy; decode advances token-by-token and is memory-bandwidth bound at practical batch sizes.
Modern optimizations form a coherent picture: FlashAttention cuts memory traffic, PagedAttention manages KV-cache allocation, GQA/MQA shrink per-token KV state, continuous batching keeps sequences mixing, chunked prefill stops long prompts from monopolizing execution, prefix caching avoids recomputation.
Scale adds complexity: tensor parallelism trades communication for memory, pipeline parallelism introduces bubbles, MoE serving adds expert routing and load imbalance; some systems split prefill and decode onto separate workers.
Tokens/sec is dangerously incomplete: production systems must track TTFT, inter-token latency, throughput, queueing, KV-cache utilization, p95/p99 latency and goodput.
Inference engineering sits at the intersection of ML systems, distributed systems, compilers, GPU architecture and performance engineering — making every GPU deliver more useful work is a discipline of its own.

Original post →

Inference engineering is the underrated AI skill: KV cache, batching and p99 latency explained

More from Infra