TURTLE paper auto-evolves tool-use benchmarks, boosting hard tasks on AppWorld by 165%
MichaelD1729 · x · 2026-10-24
The paper proposes TURTLE (Tool Use Related Tasks with Long-Horizon Evolution), an Unsupervised Environment Design approach to scale tool-use benchmarks beyond human supervision.
- Method: three LLM agents — a proposer generates new tasks from existing toolsets and task primitives, a solver executes them, and a feedback generator summarizes execution traces for the proposer to exploit; the proposer is rewarded for difficult-yet-solvable tasks, producing self-tuning test suites.
- Results: on AppWorld, τ-Bench, and AutomationBench with Gemma 4 31B-it, evolved tasks hit the target difficulty range (165% increase in hard tasks on AppWorld), increase reasoning and simultaneous tool-interaction diversity, and elicit broader failure modes for diagnosis.
By Hyunji Alex Nam, Michael Dennis, Natasha Jaques, and Minsuk Chang.
More from coding & agent
- Claude Code Routines runs overnight, screens 380 clinics into 30 outreach drafts — Arindam_1729 · 2026-10-24
- Factory and Anthropic share customer story on long-horizon coding agents and 'dark factories' — matanSF · 2026-10-24
- Dev argues Rust rewrite of Apache2 with Verus spec beats running C code in expectation — jd_pressman · 2026-10-24
- Agent-designed specs tested against every historical CVE, then adversarially attacked — jd_pressman · 2026-10-24
- Carmack amplifies Goose: a memory-safe systems language that beats C++ and Rust without a heap — ID_AA_Carmack · 2026-10-24
- Scoble points a Grok Bot at 20,000 curated accounts to build his personalized AI news feed — Scobleizer · 2026-10-24