Marin's Open Scaling Law Extrapolates 300× and Predicts Loss Within 0.2%
dlwh · x · 2026-10-27
The Marin team released Delphi, their first open scaling suite: a recipe mapping compute to model configs, a suite trained on Google's TPU Research Cloud, and a scaling law fit on 7 IsoFLOP optima (3e18–3e20 FLOPs).
Key results:
- Held-out runs at 1e21, 1e22, and 1e23 FLOPs land within 0.5% of the fit
- A pre-registered forecast predicted the largest run (1e23 FLOPs, 25B params, 600B tokens) within 0.2%, extrapolating 300× past the largest fitted run
- The first attempt diverged at 30× extrapolation; the fix was a token-horizon correction to learning rate and other hyperparameters, plus a new optimizer removing weight decay from the search
- Extrapolated, the curve places Marin 32B, Kimi K2.5, DeepSeek V4, GPT-4, Opus and GPT-5 on the same law (compute estimates via Epoch AI)
Author dlwh notes the form loss = E + A·compute^(-alpha), where estimating the asymptote E while small models are far above it is the hard part. Checkpoints, mixture, and recipe are open-sourced.
More from Research
- Kolab Benchmark Uses Pro Screen Recordings to Show AI Agents Overestimate Collaboration Quality — minnesotanlp · 2026-10-27
- One BBQ Example Pushes LLM Fairness Scores Past 90%, EMNLP Paper Questions Benchmarks — radamihalcea · 2026-10-27
- Real2Sem2Real turns one phone video into 2,000+ robot demos, matching 150 teleop demos — Scobleizer · 2026-10-27
- Understanding MuonH (Hyperball) through weight norm geometry of scale-invariant networks — jasondeanlee · 2026-10-27
- Sail equivalence proof and riscv-coq cross-checks uncover real RISC-V spec bugs — xennygrimmato_ · 2026-10-27
- Kokotajlo challenges CoT monitorability evals: what if the model didn't refuse? — dfrsrchtwts · 2026-10-27