SemiAnalysis: 4-hi HBM wins on $/bandwidth, cutting inference cost amid DRAM shortage
dylan522p · x · 2026-09-14
SemiAnalysis argues the HBM stacking race is reversing as DRAM shortages bite: next-gen accelerators will standardize on 8-hi stacks instead of today's 12-hi — Nvidia's Rubin Ultra drops from 288GB to 192GB per GPU.
The deeper claim: for bandwidth-bound inference workloads, 4-hi HBM delivers the best $/bandwidth and thus the lowest cost per token. Beyond a capacity threshold, extra stacking offers diminishing returns while carrying the same BOM penalty. Hardware teams at major labs plan to adopt this approach from HBM4 in their ASIC programs, echoing the focus on maximizing tokens/Watt under datacenter power constraints.
Related event: SemiAnalysis: HBM Stacking Trend Reverses Toward 8-hi(2 posts)→
More from Infra
- Would LITE accepting MRVL's merger offer prove its moat was overstated? — BenBajarin · 2026-10-30
- Claymation body horror on 8GB: three wrong settings, not the GPU, made local renders slow — roro9- · 2026-10-30
- durable-object-migrator: zero-downtime exports for Cloudflare Durable Objects — samgoodwin89 · 2026-10-30
- An unnamed firm already runs a fairly scaled dedicated vehicle for compute lending, says analyst — matt_slotnick · 2026-10-30
- GPU break-even throughput vs open-weight token prices up 25-83% since May — bookwormengr · 2026-10-30
- Sam Altman says the chip supply chain's invisible hand is failing — and DRAM is why — SnoozeDoggyDog · 2026-10-30