Δτ-Bench: Agents Face a Moving Tool Frontier, Best Model Scores Just 42.3%
UniversityCollegeLondon · hf · 2026-10-27
Researchers from Zhejiang University and UCL introduce Δτ-Bench, an executable agent benchmark that models the structural evolution of large tool ecosystems: independently evolving services continually reshape how tasks can be done, with familiar tools disappearing or being replaced while better alternatives emerge amid overlapping functionality and noisy docs.
- 1,588 validated multi-turn tasks across 42 domains and 6 successive ecosystem states, generated via capability-preserving transformations.
- Even the strongest model or harness achieves only 42.3% task success; initial gains from existing skill-learning methods fail to consistently translate into better adaptation as the ecosystem evolves.
- The team proposes Steward, an ecosystem manager that jointly maintains tools and procedural knowledge via incremental skill maintenance and dynamic provisioning.
The paper argues existing benchmarks fail to test whether agents can proactively discover efficient tool-use paths and promptly revise accumulated procedural knowledge — a moving capability frontier.
More from Research
- AI-designed drug works in humans: aging looks 'like a list of fixable problems' — rand_longevity · 2026-10-27
- Pessimistic complexity theory hypothesis: faster algorithms must get uglier — Elijah_Meeks · 2026-10-27
- UMAP arcing explained: one giant coordinate dominates neural network activation maps — leland_mcinnes · 2026-10-27
- NeurIPS benchmark: tree models still beat deep learning on 45 tabular datasets — tak3sh8 · 2026-10-27
- Kevin Bryan: maximizing raw accuracy is rarely the right way to train AI assistants — Afinetheorem · 2026-10-27
- Szegedy: a flaw in common training losses went unnoticed for over a decade — ChrSzegedy · 2026-10-27