SOAP/SPlus whitening enables stable pretraining across 96 DiLoCo nodes
A new arXiv paper shows that whitened update covariance optimizers like SOAP and SPlus enable stable pretraining across 96+ DiLoCo nodes, addressing compute scarcity for distributed labs.
2026-10-20 ~ 2026-10-20 · 2 related posts
- New paper shows whitening-based inner optimizers enable strong replica scaling in DiLoCo LLM pretraining — kastnerkyle · 2026-10-20
- SOAP & SPlus scale pretraining across 96+ DiLoCo workers via update whitening — aaron_defazio · 2026-10-20