NVIDIA: 3% more HBM, 30x throughput per dollar — full-stack codesign solves AI's memory wall
NVIDIA Blog · rss · 2026-11-06
NVIDIA explains how agentic inference is squeezing HBM: KV cache grows with trajectories of up to 3M tokens, and RAG adds more context, so HBM must hold both weights and cache.
Instead of just adding HBM, NVIDIA's full-stack codesign squeezes more useful work from every byte:
- Shrink weights: NVFP4 replaces FP8; models like FLUX.2 [klein] and Nemotron 3.5 Super VL now ship NVFP4 checkpoints; weights spread across up to 72 GPUs in NVL72.
- Shrink and offload KV cache: NVFP4-compressed cache; Vera CPU's large LPDDR over NVLink-C2C as a staging tier; Dynamo moves inactive context to CMX context memory storage (partners include Cloudian, DDN, Dell, HPE, IBM, NetApp, VAST, WEKA) with system-wide KV Cache Reuse.
- Keep RAG out of HBM: Nemotron Retriever embeddings plus cuVS run the index on CPU/storage, admitting only retrieved passages.
Result: from Blackwell Ultra to Vera Rubin, HBM per GPU rose only 3%, while throughput per dollar on the SemiAnalysis AgentX DeepSeek V4 Pro workload jumped up to 30x (Vera Rubin NVL72 vs GB300 at 125 tokens/s/user; GB300 was 10x vs H200). Tokens per second per dollar is rising far faster than HBM capacity.
More from Infra
- xgboost-metal hits 2.56x speedup over 18-core CPU on M5 Pro Mac mini — bingxu_ · 2026-11-06
- Neocloud bonds trade in the mid-teens, making off-taker prepayments the cheapest capital in the stack — jfischoff · 2026-11-06
- Meta et al. already eyeing data centers outside the US after local backlash — TinfoilTricorn · 2026-11-06
- Nemotron 3.5 Bolt E4B hits 27.7 tok/s on Jetson Orin Nano Super, 1.8x faster than baseline — Any_Frame9721 · 2026-11-06
- AI debt issuance hits $466B through September, over 4x last year's pace — Beth_Kindig · 2026-11-06
- Baseten launches Sandboxes, a cloud foundation built for AI agents — baseten · 2026-11-06