Groq to stack DRAM in next inference chip to overcome SRAM capacity limits
rohanpaul_ai · x · 2026-10-21
Industry sources say Groq will adopt 3D-stacked DRAM packaging in its next inference chip, moving away from fast-but-tiny SRAM. Since Nvidia paid $20B last December to license Groq's tech and hire its leaders, this is effectively Nvidia's next LPU.
Bandwidth vs capacity:
- Each generated token requires reading model weights from memory, so bandwidth sets speed
- Groq 3 LPU's 500MB on-chip SRAM runs at 150TB/s, 7x a Rubin GPU's HBM bandwidth
- But a 256-chip rack holds just 128GB vs one Rubin GPU's 288GB; an 8-bit trillion-parameter model needs 2,000 chips for weights alone
Two paths: the rumored 3DS design stacks DRAM vertically via TSV while keeping a standard DRAM interface (capacity gain at DRAM speeds), while a hybrid-bonding route (SK Hynix DRAM fused onto the logic die) promises 10x HBM bandwidth density with 77-83% less energy per bit.
Cerebras plans DRAM stacking on its CS-6 wafer too, signaling the capacity ceiling binds all SRAM-first designs. The bonding method decides the outcome.
Related event: Groq to Adopt 3D-Stacked DRAM in Next-Gen LPU to Break SRAM Capacity Limit(3 posts)→
More from Infra
- Modal open-sources Spindle, a Tinker-compatible training server you pay for in GPU time — andersonbcdefg · 2026-10-23
- Intel CTO: chip output can at best 2X while AI compute grows 3X per year — SammaVaco · 2026-10-23
- Engineer reverse-engineers projector TV's 10s wake lag to a cold-booting Linux board — banteg · 2026-10-23
- Ollama CEO says enterprise AI is having its PC moment on This Week in AI — ollama · 2026-10-23
- Modal open-sources Spindle, a Tinker-compatible training server that charges GPU time instead of tokens — josh_wills · 2026-10-23
- Aleph Alpha documents RL sandbox infrastructure lessons: stalled GPUs, full disks, crash-as-wrong-answer bugs — _ScottCondron · 2026-10-23