Andrew Perrault: eval environments released only when scorers match human majority
arnabdotorg · x · 2026-10-15
Andrew Perrault explains that his team builds programmatic scorers for each eval task. Because humans disagree substantially on these tasks, they only release environments and scorers where the scorer closely matches the human majority in their internal testing, as a quality gate for reliable evaluations.
More from Research
- New AI benchmark released, already saturated at 98% by model dubbed Astra — tszzl · 2026-10-15
- AIPOCH Open-Science launches as open-source local-first AI research workbench with reproducible .science packages — JaynitMakwana · 2026-10-15
- Mathematicians hail the "OAI proof": expository paper on no Siegel zeros goes up — basedjensen · 2026-10-15
- Rejected by NeurIPS: study shows a single test-set replica beats uncontaminated irreducible error — RylanSchaeffer · 2026-10-15
- Ofir Press's team open-sources a multimodal coding benchmark probing the capability frontier — OfirPress · 2026-10-15
- AdaBoost co-creator Robert Schapire presents 'Convex Analysis at Infinity' work, book draft now public — prof_kamilov · 2026-10-15