Coding Agents Beat Hand-Engineered Planners at Generalized TAMP, 56%-95% vs 47%
FBK-NLP · hf · 2026-09-25
FBK-NLP tested whether coding agents can automate generalized task and motion planning (TAMP): given a task description and simulator access, agents interact with the environment to synthesize a program, which is then frozen and evaluated on unseen instances.
- Agents tested: Claude Code (Opus 5) and Codex (GPT-5.6 Sol, GPT-6 Astra) across 28 simulated environments from KinDER and PDDLStream;
- 980 generated programs, 100 held-out instances each — 98,000 evaluation episodes in total;
- All three agent setups reach 56%-95% mean success vs 47% for hand-engineered planners, one-shot generation, and an LLM generalized-planning baseline;
- As object counts grow, agent programs stay ahead while using roughly an order of magnitude less compute per instance;
- Logs show agents calibrating physical models, testing edge cases, and refining strategies via interaction;
Code and full prompts are open-sourced. Coding agents are a strong baseline for generalized TAMP.
More from coding & agent
- FlexViz keeps 100M+ row charts interactive with lazy Polars aggregations and Rust kernels — JeremyCMorgan · 2026-09-26
- Built in an hour with Claude Code: the killer AI apps may be the ones we make for each other — alfred_lua · 2026-09-26
- Evals tell you it got worse, gates stop you shipping it: building an LLM gate in GitHub Actions — bgoncalves · 2026-09-26
- Building named-dimension signal vectors to cluster and search agent traces — HanchungLee · 2026-09-26
- The Claudification of SEO Audits: AI-Generated 30+ Page Reports Spreading — lilyraynyc · 2026-09-26
- Cua ships stable Omarchy Driver: OS-level multi-cursor computer use, open source — alexcovo_eth · 2026-09-26