EvalResearchBench: Can AI Agents Build Their Own Feedback for Recursive Self-Improvement?
Yaolun Zhang, Tianyi Xu, Yujie Zhao, Jishen Zhao, Qingyun Wu, Huazheng Wang
cs.AI
2026-10-03
EvalResearchBench has 9 agents build their own evaluators in 3 hours and $200; the best reproduces ~75% of target rankings vs a 91% ceiling, and human random sampling remains strong.
Recursive self-improvement (RSI) leans on evaluation feedback to judge whether a change actually helped. Running complex benchmarks on every iteration is expensive and slow, and agentic evaluations with long tool-use trajectories make it worse. Human experts cut the cost by picking representative subsets or building cheap proxies, as tinyBenchmarks does. Existing automated-improvement systems still leave that design step to humans: PostTrainBench fixes the target benchmark, RSIBench-Data uses a fixed evaluation service, AlphaEvolve evolves programs against a user-supplied function. Move to a new domain and someone has to hand-build an eval first. ERB (EvalResearchBench) isolates that step and asks whether the agent can build the measurement itself.
The setup works like a timed hackathon. A researcher agent, a frontier model inside Codex or Claude Code, receives target descriptions, reference scores for four visible candidate models, permitted task and grading resources, execution traces, and metered APIs for dev candidates. Budgets are hard-capped: three hours, $100 for researcher inference and $100 for pilot testing in separate wallets, so deliberation cannot eat the testing allowance; frozen assessment gets its own $120 cap and $30 per candidate.
Nine researcher configurations (GPT-6 Astra, Grok 4.7, Fable 5.1, Opus 5, Kimi K3, GPT-5.6 Sol, GLM-5.3, DeepSeek V4 Pro, Fable 5) each run one episode against 14 targets. On the main table Astra takes the best overall Score and Grok 4.7 the best PA at 74.87%; domain leaders split too, with Grok ahead in coding, Astra in co-work, and Opus 5 in reasoning, while coding is the weakest domain for most.
Key numbers on the matched twelve-candidate panel:
| Metric | Best | Reference |
| Overall PA | Grok 4.7: 76.07% | Random120: 68.29%; ordering ceiling 91.20% |
| Overall Score | Grok 4.7: 22.18 | Random120: 13.10; Astra: 21.35 |
| Coding Score | Grok 4.7: 22.94 | Random120: 22.92, a tie |
| Execution cost of frozen evaluator | Grok 4.7: $8.63 | Fable 5.1: $109.41 |
The 91.20% ceiling is not a capability gap: the 14 targets disagree with each other on that share of model pairs, so no single evaluator can pass it. Seven of eight agent-built evaluators beat Random120 on Score, only four on PA. Fable 5.1 tops development-target Score while Grok tops sealed-target Score, so fitting visible references does not transfer. Two details sting. Grok puts 93.87% of its co-work weight on six of twenty questions; re-scoring its saved responses with uniform weights lifts overall Score from 22.18 to 37.03, signal its own weights wasted. Removing Astra's eight spreadsheet tasks drops co-work Score by 31.94 points, and 46 of 96 responses in that component hit the output cap without a final answer. Question count drives stability: going from 10 to 90 questions cuts Astra's Score standard deviation from 6.64 to 2.14. Fable 5 exhausts its assessment budget after 20 of 39 candidate-domain runs and is dropped. Gaming is not trivial either: a constant 0.5 evaluator scores exactly 0 on Score and 50% on PA.
This turns evaluation design from a craftsman's job into a measurable agent skill, and the reading is sober. Frontier models, given visible reference scores and room to iterate, reproduce only about three quarters of target rankings; the $109 evaluator loses while the $8.63 one takes the top PA. For teams building internal evals, two cheap lessons apply directly: use more questions, and keep weight from piling onto a few items. For anyone working on RSI, this is the gate to pass first: a self-improvement loop is only as good as the instrument it reads.
The authors list concrete failure modes: frozen evaluators truncate answers, exhaust assessment budgets, and let a few items dominate a domain score. The coding targets themselves are incompatible; ProgramBench and SWE-bench Multilingual have reference means of 0.008 versus 0.876 on shared candidates, with an oracle RMSE floor of 0.303, so raw-score reproduction in that domain is structurally hard. Open questions remain: each model runs a single episode, so sensitivity to randomness is unreported; the Score leader flips from Astra to Grok when the panel goes from 13 candidates to 12; and the three-domain taxonomy plus the choice of 14 targets are themselves human judgment. Reproduction also costs real API money.