Single-Shot Benchmarks Are Dead: Evals Are Now a Compute and Protocol Game

Gauri_the_great · x · 2026-11-21

Arguing that benchmark scores are no longer just "scores" — especially after Claude's Riemann Hypothesis proof work — the author contends static single-shot benchmarking has lost its value, while eval design matters more than ever.

Scores now hinge on:

Core claim: evals are now a compute and protocol game — capped generation budgets can make a frontier model look deceptively weak, while strong verification plus ample compute changes results dramatically.

Original post →

More from Models

Models channel →