Full talk slides released: how inference engines work, KV caching to agentic loops
nutlope · x · 2026-09-12
Engineer zainhas released full slides from his talk on how inference engines actually work, covering the end-to-end lifetime of a request: engine internals, KV + prefix caching, continuous batching, paged attention, chunked prefill, sampling, and agentic loops from inside the engine. A solid systems reference for developers working with vLLM-style stacks.
Related event: Full Slide Deck Explaining How Inference Engines Actually Work Goes Public(4 posts)→
More from Infra
- Cornell Professor's Startup Makora Wins $1.75M to Make AI Faster and Cheaper to Run — srush_nlp · 2026-10-31
- Transformers.js monthly npm downloads grow 9x in a year, from 1.64M to ~15M — nicodotdev · 2026-10-31
- Agents per user doubling every 6 months: AgentWrap rearchitects runtime around 'agent as a function' — matanSF · 2026-10-31
- Anthropic's personnel costs surged 486% to $3.15B, with stock comp at 70% — rohanpaul_ai · 2026-10-31
- Dual 3090 + 32GB RAM can't run Qwen Strata low-bit quants; user mulls a $479 RAM upgrade — runcertain · 2026-10-31
- a16z: B300 GPU rentals up 106% since July while average token prices fall 58% — a16z · 2026-10-31