Fixing compute convention and fitted floor cuts scaling forecast error ~79%
dlwh · x · 2026-10-27
dlwh revised his earlier scaling-law forecast for the Hero pretraining run: adopting Paloma's compute convention (excluding output-head FLOPs) and fitting the Uncheatable loss floor from small runs yields an asymptote E of 1.20 (the old forecast fixed it at 1.4, which he calls made up). Across the same 46 Hero 4K evals, RMS error drops from 0.031 to 0.0065 nats/token, 79% lower. The compute convention matters hugely when extrapolating: output-head FLOPs are 65% of compute at the smallest rung versus 3% at 535b, and changing flops alone actually made the fit worse.
More from Research
- Kolab Benchmark Uses Pro Screen Recordings to Show AI Agents Overestimate Collaboration Quality — minnesotanlp · 2026-10-27
- One BBQ Example Pushes LLM Fairness Scores Past 90%, EMNLP Paper Questions Benchmarks — radamihalcea · 2026-10-27
- Real2Sem2Real turns one phone video into 2,000+ robot demos, matching 150 teleop demos — Scobleizer · 2026-10-27
- Understanding MuonH (Hyperball) through weight norm geometry of scale-invariant networks — jasondeanlee · 2026-10-27
- Sail equivalence proof and riscv-coq cross-checks uncover real RISC-V spec bugs — xennygrimmato_ · 2026-10-27
- Kokotajlo challenges CoT monitorability evals: what if the model didn't refuse? — dfrsrchtwts · 2026-10-27