Reddit thread: what max-context KV reservations actually cost beyond concurrency
werunm · reddit · 2026-09-18
A Reddit inference-engineering post works through what max-context KV reservations actually cost.
- The author's first instinct was that the dead region slows decode, but rejects that: nothing reads it, it's just unavailable memory.
- The real cost is that concurrency is capped by reservations instead of live tokens, so you hit the wall far below physical capacity — the core argument for paged KV.
- Open question: if workloads are uniform and requests genuinely run near max context, does paging buy anything beyond bookkeeping?
More from Infra
- Ben Bajarin argues compute is becoming fungible as hyperscalers redesign data center economics — BenBajarin · 2026-09-18
- LLM Studio: free Android app runs LLMs offline, generates images and turns your phone into an AI server — Miserable_Bird5822 · 2026-09-18
- fal made an open-source video model 35x faster — and Hollywood became its fastest-growing segment — a16z · 2026-09-18
- AMD plans ~10% price hike across GPUs, chipsets, and possibly CPUs — FullstackSensei · 2026-09-18
- Dev Builds 'Local ChatGPT in a Box' on One RTX 5090, Sharing Every Workaround Along the Way — valdev · 2026-09-18
- King Charles Meets OpenAI, Anthropic, DeepMind and Nvidia Execs on AI Safety — eyishazyer · 2026-09-18