KV cache often spills out of HBM in the agentic era, tanking effective bandwidth

AccBalanced · x · 2026-09-06

A rebuttal to the view that inference performance tracks HBM pin bandwidth alone:

The bandwidth argument only holds when the KV cache always fits in HBM. In the agentic era of long contexts and multi-turn tasks, it often doesn't.
Once capacity runs out, tokens offload to slower DRAM/NAND tiers and latency spikes.
In a tiered memory system, capacity and bandwidth are inseparable: HBM is the fast lane, and capacity is what keeps it from clogging. Cutting HBM capacity sacrifices the very bandwidth you thought you saved.

More from Infra