DeepSpeed fixes ZeRO-1/2 gradient bug across requests, upgrade to >=0.19.9
StasBekman · x · 2026-11-04
DeepSpeed maintainer Stas Bekman merged PR #8949 fixing another correctness issue in ZeRO-1/2, discovered and fixed by Sean Bell. Users should upgrade to deepspeed>=0.19.9.
- Root cause: a gradient accumulation boundary isn't necessarily an optimizer step. Callers can finish several requests with setgradientaccumulationboundary(True) while delaying engine.step() until all requests have contributed. The non-offloaded path cleared allgradtensors at every boundary, discarding earlier requests' gradients and potentially leaking a previous update's gradients into the next.
- Fix: both non-offloaded and CPU-offloaded optimizer paths in deepspeed/runtime/zero/stage1and2.py now preserve gradients until the actual step, with no config changes or app-side workarounds.
- Bekman also wonders how these bugs survived for years before being discovered.
More from Infra
- US can't build enough data centers while China can't fill the ones it has with chips — zijing_wu · 2026-11-04
- 29 to 1: Data center restriction ballot measures dominate across the US — 4KTV · 2026-11-04
- Why Musk capped Grok Bot: $15/hr GPU spot market vs $4/hr TCO funds $400B compute plan — AccBalanced · 2026-11-04
- US datacenter buildout stalls while Paraguay and Kazakhstan scoop up AI compute — ivan_bezdomny · 2026-11-04
- Crusoe takes an energy-first approach, expanding AI data centers to Iceland, Norway and Paraguay — AccBalanced · 2026-11-04
- Crusoe CEO says its Abilene AI data centers use as much water as ~10 homes — AccBalanced · 2026-11-04