Stanford & DeepMind's TURTLE evolves tool-use benchmarks into harder, self-tuning tests
rohanpaul_ai · x · 2026-11-09
A Stanford and Google DeepMind paper introduces TURTLE, an Unsupervised Environment Design method that uses three LLM agents (proposer, solver, feedback generator) to evolve existing tool-use tasks into harder but still solvable tests. On AppWorld, τ-Bench and AutomationBench with Gemma 4 31B-it, it raised the share of hard tasks on AppWorld by 165%, increased task diversity, and elicited broader failure modes—addressing the problem of static benchmarks being quickly saturated.
Related event: Stanford and DeepMind Propose TURTLE to Evolve Harder Tool-Use Benchmarks(2 posts)→
More from coding & agent
- Revolut runs coding agents for 2,000 engineers at 1/10th cost with open models — zainhas · 2026-11-09
- Future CEO of agentic orgs will be a system designer, not a McKinsey alum — GregCook2011 · 2026-11-09
- When humans become the bottleneck, lift the loop: System 1 steers, agents deliver — dotey · 2026-11-09
- Run Claude Code on Your iPhone: Rex + Tailscale Setup Guide — leslysandra · 2026-11-09
- Ex-OpenAI RLHF lead Diogo's north star: make intelligence as boring as databases — schwentker · 2026-11-09
- Grok Ghost demo shows autonomous computer-use agent driven by screen OCR — Aiden_Tech_Ai · 2026-11-09