Grok explains why DeepSeek picked DualPipe + ZeRO-1 over ZeRO-3 on 2048 H800s
TheZachMueller · x · 2026-09-23
- Grok breaks down DeepSeek's parallelism choices on 2048 H800s: cross-node EP all-to-alls hit 1:1 compute:comm, and while FSDP/ZeRO-3 + EP can reach comparable speed, ZeRO-3's extra param all-gathers add traffic that's hard to hide with limited NVLink/IB.
- DualPipe (16-way bidirectional PP) specifically overlaps dispatch/combine ops with attention/MLP in forward+backward plus PP bubbles for near-zero overhead.
- With memory opts like recompute and FP8, ZeRO-1 + PP sufficed without TP — a useful case study in large-scale training tradeoffs.
More from Infra
- Qwen 3.8 27B local coding session runs 3 days on one RTX 4090, then spews endless slashes — Tiny-Entertainer-346 · 2026-09-23
- fmgo: call Apple's on-device Foundation Models from Go with no CGO and no Swift — Super_Run_8466 · 2026-09-23
- Huawei unveils Peerium architecture: nested BSP unifies million processors into one computer — Dr_Singularity · 2026-09-23
- AI costs fall 47% per quarter, 4x faster than DNA sequencing: Epoch AI — daveholtz · 2026-09-23
- M5 Ultra LLM test: 4x faster prompt processing, but double the power draw — DigitalguyCH · 2026-09-23
- $500 of Dell OptiPlexes become a diskless netboot lab where AI agents can't brick the hardware — colinmcnamara · 2026-09-23