Marin Shares Forecast-vs-Measured Plots: Paloma Hits 0.4% RMS Error
dlwh · x · 2026-10-27
dlwh shared plots comparing two held-out forecasts of the same Hero 535B-A23B run: the original Uncheatable forecast at 1.9% RMS error, and the main Paloma forecast at just 0.4% (RMSE 0.00856 vs 0.0311 nats/token; 46 evaluations, no smoothing). He jokes that after promising in follow-ups they'd get better at forecasting, he posted the much better Paloma plot they'd had all along.
More from Research
- Kolab Benchmark Uses Pro Screen Recordings to Show AI Agents Overestimate Collaboration Quality — minnesotanlp · 2026-10-27
- One BBQ Example Pushes LLM Fairness Scores Past 90%, EMNLP Paper Questions Benchmarks — radamihalcea · 2026-10-27
- Real2Sem2Real turns one phone video into 2,000+ robot demos, matching 150 teleop demos — Scobleizer · 2026-10-27
- Understanding MuonH (Hyperball) through weight norm geometry of scale-invariant networks — jasondeanlee · 2026-10-27
- Sail equivalence proof and riscv-coq cross-checks uncover real RISC-V spec bugs — xennygrimmato_ · 2026-10-27
- Kokotajlo challenges CoT monitorability evals: what if the model didn't refuse? — dfrsrchtwts · 2026-10-27