Single-Shot Benchmarks Are Dead: Evals Are Now a Compute and Protocol Game
Gauri_the_great · x · 2026-11-21
Arguing that benchmark scores are no longer just "scores" — especially after Claude's Riemann Hypothesis proof work — the author contends static single-shot benchmarking has lost its value, while eval design matters more than ever.
Scores now hinge on:
- whether the model can iterate across attempts
- whether it receives verifier feedback (compiler, test suite, Lean kernel)
- how compute is split between deep serial trajectories and parallel multi-agent search
Core claim: evals are now a compute and protocol game — capped generation budgets can make a frontier model look deceptively weak, while strong verification plus ample compute changes results dramatically.
More from Models
- Intel researcher: open-source models are good enough for most AI tasks — JFPuget · 2026-11-21
- GPT-6.2 Astra ranks 2nd on raw intelligence but last on writing vibes, 5-model comparison shows — iruletheworldmo · 2026-11-21
- Inkling 2 Mini hits 613 tok/s decode and 1M context on one RTX 5090 via Basalt — jesdga95 · 2026-11-21
- OpenAI releases prompt templates to fix GPT-6.2 Astra and Sol quirks — The Decoder · 2026-11-21
- METR Extends Time-Horizon Benchmark: Opus 5.5 Hits 80% Success at ~20-Hour Tasks — ben_j_todd · 2026-11-21
- Unverified leak: 32B, 122B and 256B open-weight models incoming for local AI — HankYeomans · 2026-11-21