Inverted Task Synthesis: 37K Long-Horizon Tasks at $0.05 Each, DeepSeek Pass Rate Drops to 2.5%
teortaxesTex · x · 2026-08-07
Building solvable and cheap synthetic long-horizon terminal tasks is challenging. Researchers propose an inverted approach: instead of writing the task first, they build and run the reference solution and its runtime environment, then fit a verifier and task statement to it. Tasks are solvable by construction with built-in reward signals, and verified tasks seed the next generation.
Over 15 recursive rounds, this method generated 37,484 verified long-horizon tasks at a cost of just $0.05 per task. In evaluations, DeepSeek-V4-Pro's pass@4 plummeted from 90% in Round 1 to 2.5% in Round 15. Trajectories reach 100+ steps, becoming brutally harder without breaking the generation factory.
Training with PPO on this dataset improved Qwen3.5-27B's performance on TB2 from 41.2 to 49.4. The paper, data, trajectories, and models are fully open-sourced.
Related event: Tencent's RST Framework Recursively Synthesizes Agentic Tasks at Low Cost(3 posts)→
More from coding & agent
- Simon Willison: code mode is the future of tool calling, if it can be sandboxed — springrod · 2026-10-16
- Agent named 'Ilya' reportedly scores 1.1pt higher on benchmarks — daniel_mac8 · 2026-10-16
- Gemini CLI fix: substring matching inlined unrequested binary files into model context — priyanshu1976 · 2026-10-16
- Building AI agents on real healthcare claims data: the domain work that makes them work — alex_verem · 2026-10-16
- OpenAI stealth-testing mysterious internal model 'Aeon' in Codex desktop app and Dots — cedric_chee · 2026-10-16
- Vibe-coded iPhone Duo sniping bot built with Codex in 17 minutes, 41 tests passing — xiaohu · 2026-10-16