Agents scoring low on research benchmarks may not be the model's fault: audits need realistic upper bounds

nathanbenaich · x · 2026-11-18

Abundant Research Team argues that low scores on research-agent benchmarks (MLE-bench, RE-Bench, PaperBench, LifeSciBench, etc.) often reflect task design rather than model weakness: research tasks have no known correct answers, and noisy data or missing information creates hard ceilings even for ideal agents. They adopt RE-Bench's approach of scoring 0 for a naive baseline and 1 for a strong reference solution, and show that overly high targets compress rewards toward 0. Every benchmark needs an audited realistic upper bound to tell whether an agent is truly underperforming.

Original post →

More from coding & agent

coding & agent channel →