Eval scores reported to 5 significant digits despite 2-5 point error bars, researcher flags
dfrsrchtwts · x · 2026-09-23
A researcher calls out eval reporting that presents AECI scores to two decimal places (5 significant digits) even though error bars span 2-5 AECI points — meaning the 4th, and arguably even the 3rd, significant digit is meaningless. A pointed critique of false precision in current model evaluations.
More from Research
- ICLR load debate: capping submissions per author would barely dent paper counts, data shows — jonasgeiping · 2026-09-23
- Trust AI R&D evals only if scored by people who've hand-labeled outputs — dfrsrchtwts · 2026-09-23
- JevBench Model Decision Index Debuts on Hugging Face, Native Image Support Still Missing — openSourcerer9000 · 2026-09-23
- AI swarms spontaneously grow scale-free hub topologies, MIT professor observes — ProfBuehlerMIT · 2026-09-23
- Flex-π: a 6B world-action model beats π0.5 by up to 6x on real bimanual tasks — chris_j_paxton · 2026-09-23
- Deoptimization Is Harder Than Optimization: How JIT Escape Hatches Keep Dynamic Languages Fast — lauriewired · 2026-09-23