Stanford and DeepMind evolve stale benchmarks into hard tests with TURTLE, targeting 50% solver success
rohanpaul_ai · x · 2026-11-09
A new Stanford and Google DeepMind paper introduces TURTLE, which uses AlphaEvolve to mutate existing tool-use benchmark tasks (like AppWorld) instead of building new benchmarks from scratch.
- How it works: A proposer adds filters, extra steps, and hidden lookups to existing tasks; a solver model runs each evolved task 10 times, and tasks are kept when the solver succeeds 50% of the time (each must be solved at least once).
- Results: 69% of original AppWorld tasks were always solved by Gemma 4 31B-it, while nearly all 735 evolved tasks landed in the 40%–60% completion range. An LLM judge rated 52.5% of evolved tasks as hard, versus 19.8% of the originals.
- Takeaway: If your model aces a benchmark, evolve its tasks toward a 50% success rate rather than writing a new benchmark.
Related event: Stanford and DeepMind Propose TURTLE to Evolve Harder Tool-Use Benchmarks(2 posts)→
More from coding & agent
- Store catalog API talks a shopping agent out of a $149 sale, no human in the loop — schwentker · 2026-11-09
- Vork: an agent-native 3D engine that renders 24x faster than Blender with a 2,500-token API — ravisparikh · 2026-11-09
- Vork engine in action: Opus in Claude Code renders 1,000 Clawds in ~2 hours on an RTX 3090 — ravisparikh · 2026-11-09
- Out of Claude and Codex tokens, dev spins up local vLLM serving Qwen FP4 model for agents — alfcnz · 2026-11-09
- iamtrask says he's building agent-to-agent discovery and map-reduce protocols — iamtrask · 2026-11-09
- AGI House's personal agents build day pushes local inference and intelligence-per-watt — agihouse_org · 2026-11-09