Inference Engineering Is the Most Underrated Skill in AI, Long Thread Explains Why
techNmak · x · 2026-11-18
techNmak published a long thread arguing inference engineering is becoming one of the most important—and underestimated—skill sets in AI.
Key points
- Training gets the attention, but serving real traffic surfaces a different problem class: queueing, batching, KV-cache pressure, GPU memory, scheduling, and tail latency.
- Prefill vs decode: prefill is parallel-friendly and compute-bound; decode is autoregressive and memory-bandwidth-bound at practical batch sizes.
- Each optimization targets a specific issue: FlashAttention cuts attention memory traffic, PagedAttention reduces KV fragmentation, MQA/GQA shrink KV heads, continuous batching and chunked prefill improve scheduling, prefix caching saves prefill only.
- Memory, quantization, speculative decoding, tensor/pipeline parallelism, MoE routing, and prefill/decode disaggregation all add their own costs and tradeoffs.
- Tokens per second isn't enough: production systems need TTFT, ITL/TPOT, goodput, KV occupancy, and tail latency; high aggregate throughput can still mean users waiting too long for the first token.
Includes a full infographic covering the serving stack, key metrics, roofline analysis, memory formulas, and fast diagnosis.
More from Infra
- Turn an 8GB old laptop into a private offline AI in 20 minutes with Jan and Gemma — alex_verem · 2026-11-18
- Deep dive: the full rise and fall of DEC's Alpha processor architecture — ArtificialOther · 2026-11-18
- Nightmare Mobile Runs FLUX, Qwen Image and Wan 2.2 Fully Offline on Phone NPUs — Few_Caregiver8134 · 2026-11-18
- Free online book: LLM Inference Handbook for production deployment — burkov · 2026-11-18
- SEC opens civil probe into Nvidia over undisclosed China-linked chip sales — The Decoder · 2026-11-18
- Open-source LLM Optimization Handbook: 48 topics with hands-on quantization, KV cache and kernel problems — udmrzn · 2026-11-18