Microsoft's ProgramDistill: 4,063 tasks expose coding agents' interactive blind spots
MicrosoftResearch · hf · 2026-09-17
Microsoft Research releases ProgramDistill, a benchmark evaluating coding agents on inferring behavior from working reference applications rather than following instructions. Its mine-craft-patch pipeline mines 1,975 replay-verified behaviors across 26 apps, yielding 4,063 tasks with no human intervention. Among nine frontier agents, GPT-6 Astra and Claude Opus 5 reach 49.2% and 28.8% success on cumulative full-application reconstruction; in partial reconstruction, success drops from 100%/96% to 64%/32% as restoration depth goes from 1 to 8.
More from coding & agent
- General Reasoning explains how to train agent swarms with RL and latency pressure — rosstaylor90 · 2026-11-02
- "Coding isn't the bottleneck anymore — usage windows are," says founder Kyle Gawley — kylegawley · 2026-11-02
- Demo passed ≠ production-ready: 5 questions to answer before letting agents run your business — _kellhus · 2026-11-02
- An AI agent passed 100 test chats—still not ready to run the business — djgoel123 · 2026-11-02
- Agent given total creative freedom produces a 1:44 code-drawn animated short film — ivegotnoidea1 · 2026-11-02
- Meme: Wake Up, Use Claude Code, Repeat All Day — tekbog · 2026-11-02