Independent check verifies Horizon's SOTA claims on held-out synthetic benchmarks
maksym_andr · x · 2026-11-19
Skeptical that Opus 5.5 overclaimed its contributions, the team manually verified performance on newly generated synthetic datasets unseen by the models — and found them genuinely good. Their report found no evidence of overfitting or benchmark gaming, noted Horizon picks checkpoints by validation F1 (a better proxy even when reported test scores look worse), and using MiMo V2.6 Pro-generated held-out benchmarks across 4 model families found the models often SOTA for their size on narrow tasks, e.g. Injection Guard variants on the Pareto front.
More from Models
- Gemini 4 Neon launch sweeps APEX-Agents leaderboard, reignites Google's frontier race — krishnan · 2026-11-19
- Gemini 4 Neon hits #2 on Vals Index at $7.40/task, near top of price-performance frontier — OfficialLoganK · 2026-11-19
- Gemini 4 Neon debuts at #2 on Vals Index at less than half Claude's cost — demishassabis · 2026-11-19
- Gemini audits 30k lines written by Opus 5.5, finds 29 real bugs with zero false positives — neurocy · 2026-11-19
- Top 10% OpenAI researchers reportedly burn $25,000 of tokens a day — ivan_bezdomny · 2026-11-19
- Claude Max for $29 instead of $200: reseller gray market sparks debate — mehdiweb · 2026-11-19