Private AI benchmarks start ~11 points harder but saturate just as fast as public ones
emollick · x · 2026-11-15
Ethan Mollick shares Benchmark Almanac data (26 private/held-out vs 146 public SatBench benchmarks, medians): private benchmarks start 11 points lower (best AI score 53 vs 64 three months after release) but climb just as fast — +21 vs +18 points in the following year, and +24 vs +13 points per year from release to today. Fewer private benchmarks have crossed 85% simply because they start harder. Data as of Nov 14, 2026, including Epoch AI Benchmarking Hub.
Related event: SatBench Tracks AI Benchmark Saturation: 63% of 173 Tests Maxed Out(3 posts)→
More from Models
- Gemini math lead reveals inside story: evals, post-training recipes, scaled runs — thtrieu_ · 2026-11-15
- Blogger saves $53 in 43 hours after switching from Claude Code to Gemini 4 Carbon — mazzaTalk · 2026-11-15
- SatBench: 63% of lab-cited AI benchmarks already saturated, and solving is speeding up — emollick · 2026-11-15
- LFM2.5-24B-A2B praised for local agent use: 37 tok/s on an AMD 780M iGPU — jeremyckahn · 2026-11-15
- Will Depue's Blackwell 'forensic evidence' claim on DeepSeek hits snag: V4.1-Flash not cracked — teortaxesTex · 2026-11-15
- nanoGPT Speedrun Gains Mostly From Environment Design; Newer Models Beat Human Record — eliebakouch · 2026-11-15