Memory wall chart: compute up ~3x per two years, HBM bandwidth under 2x
Summit-Star001 · reddit · 2026-09-14
From Raghu Sreeramanenei's memory tutorial at Hot Chips 2026: normalized compute (TPU v3 through R200) grows 3x every two years while HBM bandwidth (HBM2e through HBM4) grows under 2x — and the log scale hides the widening gap. Three fixes are in flight: memory beside compute, memory closer on shorter links, and multiply units inside memory itself. Samsung's approach already ships in LPDDR5X, measuring 3.01x tokens/sec on Llama 3.1 8B.
More from Infra
- Your AI agent would cost 260% more without prompt caching, Arena data shows — arena · 2026-10-30
- Nebius runs auction-driven GPU pricing, lifting rates ~20% amid tight supply to 2028 — BenBajarin · 2026-10-30
- macOS update silently re-adds eMudhra root certs users had removed — steipete · 2026-10-30
- Bittensor's Terza Inference routes one API across miners' models on subnet 103 — const_reborn · 2026-10-30
- SGLang Summit 2026 sells out sponsor booths and Full Passes ahead of Nov 12–13 event — ying11231 · 2026-10-30
- OpenRouter rebuilds provider routing: 46% fewer capacity errors, 16.9% faster TTFT — steipete · 2026-10-30