Modular's LLM Inference Handbook covers prefill, KV cache, batching, quantization and speculative decoding
techNmak · x · 2026-09-04
A recommended resource: Modular's LLM Inference Handbook bridges "how LLMs work" and "how LLM systems serve requests," covering prefill, decode, KV cache, batching, scheduling, quantization, prefix caching and speculative decoding.
It also points to Maarten Grootendorst's Visual Guide to Mixture of Experts, which explains routers, experts, sparse activation, token routing and load balancing with dozens of visuals.
Related event: Stop learning AI from static diagrams: a curated visual learning roadmap(13 posts)→
More from Infra
- Railway pauses all new builds amid major outage, users ask if vibe coding is to blame — AaronBergman18 · 2026-10-21
- a16z: Superhuman AI hardware design is the easy part — manufacturing is the hard one — a16z · 2026-10-21
- Sizing a 1 GW datacenter: ~522,000 Blackwell GPUs fit inside — bookwormengr · 2026-10-21
- DeepSeek's production density reportedly hits ~$38 per GPU-hour at saturation — bookwormengr · 2026-10-21
- PyTorch details KTIR and Dataflow Scheduler behind IBM's Spyre accelerator backend — PyTorch · 2026-10-21
- Indie dev replaces $1k OpenAI bill with a YAML pipeline on spot GPUs and vLLM — cmm324 · 2026-10-21