Researcher flags false precision in AECI scores and saturated AI R&D benchmarks

dfrsrchtwts criticized how a model evaluation report presents its results: AECI scores are reported to two decimal places (about 5 significant figures), yet error bars span 2–5 AECI points, so the 4th — even 3rd — significant figure carries no statistical meaning; given the tiny gaps between models and the fact that scores can shift wholesale without changing meaning, such over-precision is dishonest.

Confirmed

The criticism targets how the report presents AECI scores: two decimal places, 5 significant figures, against error bars of 2–5 points.
A dedicated section of the report argues that AI R&D-style evaluations have saturated and no longer carry information, a judgment dfrsrchtwts endorses.
In follow-up discussion, the researcher clarified the criticism targets non-METR AI R&D evals; METR evals like Sunlight and Budget NanoGPT belong to a different system.

Why it matters

This is a representative case of criticism aimed at today's overly precise, error-blind model evaluation reporting: when error bars dwarf the reported precision, trailing digits convey false certainty.
A frontier lab (Anthropic, in this case) admitting in its own report that AI R&D-style evals have saturated and lost discriminative power reflects labs' self-questioning about their capability-evaluation frameworks — and hints the space needs new eval designs.

2026-09-23 ~ 2026-09-23 · 5 related posts

Primary sources

Eval scores reported to 5 significant digits despite 2-5 point error bars, researcher flags — dfrsrchtwts · 2026-09-23 01:17 (UTC+08:00)
Researcher: non-METR AI R&D evals are saturated and uninformative — dfrsrchtwts · 2026-09-23 01:20 (UTC+08:00)

Anthropic report argues AI R&D evals are saturated and uninformative — dfrsrchtwts · 2026-09-23
AI R&D evals are saturated and uninformative, and 5-significant-digit scores don't help — dfrsrchtwts · 2026-09-23
[source] Eval scores reported to 5 significant digits despite 2-5 point error bars, researcher flags — dfrsrchtwts · 2026-09-23
[source] Researcher: non-METR AI R&D evals are saturated and uninformative — dfrsrchtwts · 2026-09-23
AECI scores reported to two decimals, but the 5th significant digit is likely noise — dfrsrchtwts · 2026-09-23