Inverted Task Synthesis: 37K Long-Horizon Tasks at $0.05 Each, DeepSeek Pass Rate Drops to 2.5%
teortaxesTex · x · 2026-08-07
Building solvable and cheap synthetic long-horizon terminal tasks is challenging. Researchers propose an inverted approach: instead of writing the task first, they build and run the reference solution and its runtime environment, then fit a verifier and task statement to it. Tasks are solvable by construction with built-in reward signals, and verified tasks seed the next generation.
Over 15 recursive rounds, this method generated 37,484 verified long-horizon tasks at a cost of just $0.05 per task. In evaluations, DeepSeek-V4-Pro's pass@4 plummeted from 90% in Round 1 to 2.5% in Round 15. Trajectories reach 100+ steps, becoming brutally harder without breaking the generation factory.
Training with PPO on this dataset improved Qwen3.5-27B's performance on TB2 from 41.2 to 49.4. The paper, data, trajectories, and models are fully open-sourced.
Related event: Tencent's RST Framework Synthesizes Long-Horizon Tasks at Low Cost(2 posts)→
More from coding & agent
- Claude Code Introduces Self-Hosted Runners for Web/Mobile Sessions — ClaudeCodeLog · 2026-08-07
- Claude Code CLI update: Self-hosting and cross-session permission controls — ClaudeCodeLog · 2026-08-07
- Claude Code gets 31 updates including self-hosted runner — ClaudeCodeLog · 2026-08-07
- swyx Offers $10k Bounty for Weekend AI Hackathon to Clone Enterprise SaaS — brandon_galang · 2026-08-07
- Qwen-CUA: A Pure Vision-Based General Computer Use Agent — rohanpaul_ai · 2026-08-07
- Integrating AI into ASP.NET Core: Patterns and Microsoft AgentFramework — plakhlani · 2026-08-07