DeepSpeed fixes ZeRO-2 fp32 precision bug, users urged to upgrade to 0.19.8+
StasBekman · x · 2026-10-30
DeepSpeed maintainer Stas Bekman issued a PSA: ZeRO-2 users should upgrade to deepspeed>=0.19.8.
- The bug: with bf16 models configured with gradaccumdtype: fp32 and gradient accumulation steps (GAS) >1, ZeRO-2 could still store the reduced gradient partition in bf16, rounding away fp32 precision after communication.
- The fix: ZeRO-1/2 now use fp32 partition storage under fp32 accumulation, while shaving 2 bytes per parameter. A benchmark with a 1B-param bf16 model on 2 H200 ranks shows peak reserved memory dropping 2,048 MiB, enabling larger models or longer sequences.
- The fix was merged as PR #8749.
More from Infra
- Cerebras breaks down the math of deploying multi-trillion parameter models at scale — ceciletamura · 2026-10-30
- Nscale's unproven CEO bids for $30 billion IPO to sell investors on his neocloud — dinabass · 2026-10-30
- Crusoe closes round at $31B valuation, from Bitcoin mining to AI data centers — nathanbenaich · 2026-10-30
- AWS hits $184B ARR, up 39.3% year over year — RihardJarc · 2026-10-30
- AI boom: US data center & computer spending now 50% larger than home construction — HaydnBelfield · 2026-10-30
- Whatever Happened to SanDisk's High Bandwidth Flash? 3TB Per GPU Promise Met With Silence — Zombiecidialfreak · 2026-10-30