U. Tokyo Study: 70% of AI Agent Tests Are Wasteful, New Method Cuts Costs by 80%
rohanpaul_ai · x · 2026-08-23
A new study from the University of Tokyo argues that most test sets used to grade AI agents are a waste of money. The research reveals that over 70% of tasks in typical eval sets consist of problems that every version passes (too easy) or none can pass (too hard), offering no signal despite high costs.
To address this, the researchers propose Task-CoEvolve. This method selects only the "borderline" tasks where past versions disagreed, dynamically updating the test set each round. It also mathematically adjusts scores to ensure comparability across rounds with different tests.
Experiments on Terminal-Bench 2.1 show:
- Grading on just 20% of the 89 tasks yields results within about one task of grading on the full set.
- Costs drop by 67-80%, and time is halved.
- While random selection is cheaper, it scores 3.3 points worse by picking easy tasks, proving that "cheap and useful are not the same thing."
Paper: Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection
More from Research
- Screenwriting principles from 'Save the Cat' apply to ML papers — OfirPress · 2026-08-23
- AI uses its own brainstorming skill to autonomously generate art — cocktailpeanut · 2026-08-23
- Study finds AI agents lock in training strategies early, hindering recursive self-improvement — omarsar0 · 2026-08-23
- Steganographic communication may emerge in multi-agent RL without obfuscation rewards — brianryhuang · 2026-08-23
- First personalized mRNA cancer vaccine wins Phase 3 trial — rand_longevity · 2026-08-23
- Explore Unsaturated Benchmarks: Frontierswe and More Reveal Model Shortcomings — dejavucoder · 2026-08-23