SOAP & SPlus scale pretraining across 96+ DiLoCo workers via update whitening
aaron_defazio · x · 2026-10-20
Aiming at labs that can't buy enough co-located chips from a single provider, new work shows SOAP and SPlus optimizers achieve strong worker scaling to 96+ DiLoCo workers by whitening the update's covariance. This suggests frontier-scale pretraining can run across heterogeneous, non-co-located compute from multiple providers. Details in a 9-part thread.
Related event: SOAP/SPlus whitening enables stable pretraining across 96 DiLoCo nodes(2 posts)→
More from Infra
- RTX 5090 Laptop GPU hangs with Xid 8 under Linux — CUDA graphs bug still unfixed — SOC_FreeDiver · 2026-10-20
- Agent inference costs spiral as GPU supply tightens — devs compare cloud, self-hosted, decentralized options — aruthesa · 2026-10-20
- Taco: hardware-agnostic LLM compression with one hash across 7 backends — phoenix10701 · 2026-10-20
- Scaling past 1T params: routing, hybrid attention, and 1M-token contexts — philipkiely · 2026-10-20
- NVIDIA partners with Germany on industrial AI, backs four German startups — nvidia · 2026-10-20
- DiffusionGemma's blazing local speeds win fans for offline image-prompt agent — SysPsych · 2026-10-20