Modular's LLM Inference Handbook covers prefill, KV cache, batching, quantization and speculative decoding
techNmak · x · 2026-09-04
A recommended resource: Modular's LLM Inference Handbook bridges "how LLMs work" and "how LLM systems serve requests," covering prefill, decode, KV cache, batching, scheduling, quantization, prefix caching and speculative decoding.
It also points to Maarten Grootendorst's Visual Guide to Mixture of Experts, which explains routers, experts, sparse activation, token routing and load balancing with dozens of visuals.
Related event: Stop learning AI from static diagrams: a curated visual learning roadmap(13 posts)→
More from Infra
- Developer Moves AI Agents to a Dedicated Linux Workstation Using Incus System Containers — kevinkern · 2026-10-21
- Nvidia guides 70% growth citing unmet AI demand, while media panics over a $500M Korean neocloud IPO — firstadopter · 2026-10-21
- Pichai: everyone is massively short on AI compute — 'everyone wishes you had two times what you had planned' — firstadopter · 2026-10-21
- Stanford CS336 Lecture 6: fused Triton kernels cut softmax memory traffic ~4x — stanfordnlp · 2026-10-21
- General Compute partners with Positron AI: Atlas memory-centric hardware ships Q1 2027 — aftahi_ai · 2026-10-21
- AI bond issuance hits $419B in 2026, up 90% from all of 2025 — SumitGup · 2026-10-21