One-figure paper: the best optimizer depends on batch size, SOAP and Shampoo swap ranks
ricklamers · x · 2026-10-28
ricklamers highlights the key figure of arXiv paper abs/2610.08975: on the Modded-NanoGPT benchmark with a fixed 1.7B-token budget and per-batch-size tuned optimizers, validation loss shows "the best optimizer depends on batch size." SOAP wins from 128K to 1M (by up to 0.012 nats) but drops to third at 2M, 0.021 nats behind Shampoo; Muon and Shampoo also cross over (Muon ahead by 0.0035 at 512K, Shampoo by 0.0037 at 2M). Directly relevant to LLM pretraining tuning.
More from Research
- CVPR 2027 Submissions Open: Registration Due Nov 10, Updated LLM-Use and Reciprocal Reviewing Policies — humphrey_shi · 2026-10-28
- Agent Swarm Formalized the 15,000-Page Finite Simple Group Classification in 24 Days — Verification, Not Generation, Was the Bottleneck — docdavkitty · 2026-10-28
- RWE-Bench launches to eval AI on real-world engineering: rockets, engines, chips — Jsevillamol · 2026-10-28
- Idempotency keys for AI agents belong in your code, not the model — shoumikgoswami · 2026-10-28
- Ben Lorica Changes His Mind on World Models as Agents Start Writing Robot Code — bigdata · 2026-10-28
- Neocambrian's from-scratch NeoSLAM tops LaMAria Long category among mono+IMU methods — AashaySachdeva · 2026-10-28