New paper shows whitening-based inner optimizers enable strong replica scaling in DiLoCo LLM pretraining
kastnerkyle · x · 2026-10-20
A new arXiv paper from Eugene Belilovsky's lab, "How Inner Optimizers Shape Replica Scaling," studies local optimization methods like DiLoCo for pretraining LLMs across poorly interconnected compute pools.
Key findings:
- Increasing model replicas under a fixed training budget is known to monotonically hurt performance; prior work showed the inner optimizer choice (Muon vs AdamW) matters but with limited explanation.
- The paper identifies two factors: the covariance structure of pseudo-gradients and the effect of batch size on pseudo-gradient noise scale.
- Theoretically and empirically, whitening-based preconditioning is a key desideratum: whitening inner optimizers achieve strong replica scaling across 8–96 replicas, with gains over data parallelism growing as parameters and data scale.
More from Infra
- REAX inference engine runs Perplexity Decider up to 6x faster, free for individuals — airesearch12 · 2026-10-20
- REAX engine to be continuously improved via open engineering competition — airesearch12 · 2026-10-20
- BitNet CPU inference is silently wrong: perplexity 6x reference, one-line fix works — matheus-c-feitosa · 2026-10-20
- Data centers are the real paperclips and the global bond market is the steel — m2saxon · 2026-10-20
- AMD: World now consumes 40 quadrillion inference tokens monthly, up 95x in two years — Beth_Kindig · 2026-10-20
- Meta details Prometheus gigawatt-scale supercluster, targeting ~1M GPUs — BenBajarin · 2026-10-20