U. Tokyo Study: 70% of AI Agent Tests Are Wasteful, New Method Cuts Costs by 80%
rohanpaul_ai · x · 2026-08-23
A new study from the University of Tokyo argues that most test sets used to grade AI agents are a waste of money. The research reveals that over 70% of tasks in typical eval sets consist of problems that every version passes (too easy) or none can pass (too hard), offering no signal despite high costs.
To address this, the researchers propose Task-CoEvolve. This method selects only the "borderline" tasks where past versions disagreed, dynamically updating the test set each round. It also mathematically adjusts scores to ensure comparability across rounds with different tests.
Experiments on Terminal-Bench 2.1 show:
- Grading on just 20% of the 89 tasks yields results within about one task of grading on the full set.
- Costs drop by 67-80%, and time is halved.
- While random selection is cheaper, it scores 3.3 points worse by picking easy tasks, proving that "cheap and useful are not the same thing."
Paper: Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection
More from Research
- U of T team mines 122,194 patient reviews across 45 Ontario hospitals with NLP to expose satisfaction and equity patterns — ishtiaqueSIA · 2026-10-16
- University of Toronto hiring AI postdoc: builders of interventions, not just benchmarkers, due Oct 30 — ishtiaqueSIA · 2026-10-16
- WorldMemArena Accepted at NeurIPS: Memory Storage ≠ Memory Use — xwang_lk · 2026-10-16
- Mech Interp Veteran Rethinks Alignment, Misgeneralization, CoT Oversight — akbirthko · 2026-10-16
- WEDGE paper generates training environments conditioned on navigation agent failures — _akhaliq · 2026-10-16
- RExBench: best coding agent completes only 33% of autonomous research extensions — najoungkim · 2026-10-16