Modular handbook maps the hidden costs of LLM inference, from KV cache to prefill/decode splits
udmrzn · x · 2026-07-28
What the handbook covers
Modular’s LLM Inference Handbook is a practical reference for deploying and scaling LLMs in production. It walks through the main performance metrics — time to first token, tokens per second, throughput and goodput — and explains why inference often becomes the bottleneck once you move from demos to real workloads.
Key topics
- Model and GPU selection
- Quantization
- Continuous batching
- KV cache growth and memory planning
- Speculative decoding
- Prefill/decode disaggregation
- Distributed deployment patterns such as BYOC and on-prem
It also includes interactive calculators and visual tools to help estimate memory usage, understand how context length affects KV cache, and see why concurrency quickly shifts the system bottleneck from model execution to queueing, latency, and memory pressure.
Related event: Modular Releases LLM Inference Handbook to Optimize Deployment(2 posts)→
More from Infra
- Gemma 4 is benchmarked locally on a 48GB Mac with MLX, llama.cpp and Java 25 — rseroter · 2026-07-28
- Bittensor subnet expansion is pitched as a cheaper AI infrastructure path for companies — markjeffrey · 2026-07-28
- Project Orion is training a 16B model live across three continents on heterogeneous compute — markjeffrey · 2026-07-28
- Kimi K3 reaches Merge Gateway with U.S. inference providers and ZDR terms — shensi · 2026-07-28
- Compute, not algorithms, is the real moat in frontier AI — GavinSBaker · 2026-07-28
- Renting GPUs and open-weight models cut one AI bill from $1.2M to $100K — kimmonismus · 2026-07-28