Agents scoring low on research benchmarks may not be the model's fault: audits need realistic upper bounds
nathanbenaich · x · 2026-11-18
Abundant Research Team argues that low scores on research-agent benchmarks (MLE-bench, RE-Bench, PaperBench, LifeSciBench, etc.) often reflect task design rather than model weakness: research tasks have no known correct answers, and noisy data or missing information creates hard ceilings even for ideal agents. They adopt RE-Bench's approach of scoring 0 for a naive baseline and 1 for a strong reference solution, and show that overly high targets compress rewards toward 0. Every benchmark needs an audited realistic upper bound to tell whether an agent is truly underperforming.
More from coding & agent
- Pawel Huryn open-sources 45 agent skills distilled from shipping three live products — PawelHuryn · 2026-11-18
- Google donates Agent Substrate to CNCF: 500ms restores, 10x density — krishnan · 2026-11-18
- Code model eval: Rust hits 95-100% accuracy as 13 languages see big gains — shuding · 2026-11-18
- Law school AI course wraps: 6 student teams ship working legal AI prototypes — jkubicki · 2026-11-18
- n8n's open-source agent Nod: secure remote access via private reverse proxy — victorialslocum · 2026-11-18
- Floot Design launches to kill AI slop: pick your UI before any code ships — heyshrutimishra · 2026-11-18