ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks
Jeonghye Kim, Minseon Kim, Young Jin Kim, Matheus Pereira, Marc-Alexandre Côté, Alessandro Sordoni, Xingdi Yuan, Zhengyan Shi
cs.SE, cs.AI
2026-09-16
A pipeline turns 26 live web apps into 4,063 replay-verified repairs. Astra falls from 100% at depth 1 to 64% at depth 8; full-app workflows hit 49.2%.
SWE-bench-style evals assume the desired behavior is already written down as an issue or a test. Real web work often runs the other way: a clickable reference (old version, prototype, sibling product) sits next to an incomplete repo, and the developer has to exercise the reference, infer the behavior, then patch the code. ProgramBench asked agents to reconstruct compiled programs from I/O; the unit was the whole program. Interactive apps add prerequisite chains (log in before creating an object, create before edit). Treating the app as one reconstruction blob erases that grain.
ProgramDistill factorizes a running web app into replayable behaviors, then composes them along prerequisite lineages from one-feature repair to full-app reconstruction. The work is from KAIST and Microsoft Research Montréal. Nine frontier coding agents are scored, including GPT-6 Astra, Claude Opus 5, and Grok 4.6.
The pipeline is mine-craft-patch. GPT-5.6 Sol drives every construction stage.
Twenty-six apps come from the OSWorld web suite and open-source SaaS clones. A shared deterministic clock and state reset make replay possible. The browser helper is Playwright-based. Evaluated agents get structured observations only, no screenshots.
The pipeline admits 1,975 replay-verified behaviors and 4,063 tasks (2,862 atomic, 1,201 cumulative). A stratified 300-task suite, ProgramDistill-300, covers depths 1 through 8 and all 26 apps.
Partial reconstruction (repair a masked repo), mean binary success:
| Model | Binary success | Cost / trajectory |
| GPT-6 Astra | 84.3% | $33.99 |
| Claude Opus 5 | 68.7% | $28.86 |
| GPT-5.6 Sol | 60.7% | $16.01 |
| Grok 4.6 | 48.3% | $8.83 |
| Claude Sonnet 5 | 47.3% | $13.12 |
| GPT-5.3 Codex | 45.7% | $5.99 |
| Gemini 3.7 Flash | 45.3% | $3.89 |
At depth 1, Astra is 100% and Opus 96%. At depth 8 those numbers are 64.0% and 32.0%. Logic-only is easier than logic-and-UI (Astra 92.9% vs 76.9% binary). Astra watches more and edits less: about 96.3 current-app observation steps per trajectory and 9.9 edits. From depth 1 to 8, lines to restore grow about 9.3x, while reference observations per target fall from 34.60 to 8.46. Shortcut attempts (Git history, public source) appear in 7.2% of trajectories: 19.0% for Grok 4.6, 0.3% for Astra.
Full-app reconstruction starts from a minimal scaffold on 12 stateful apps. Astra recovers 58.98% of atomic behaviors and 49.15% of cumulative workflows; Opus 42.03% / 28.81%; Sol 33.39% / 21.07%. Among failures, 59.2% were never observed in the reference; 27.9% were observed but ended in the wrong state, route, or result.
This is the closest replay-verifiable stand-in for "fix the web app by using the live product." The same trace is the spec and the test. Restoration depth splits "can patch one control" from "keep an eight-step workflow faithful," and the second is the cliff. Trajectories also show that leaderboard gaps are partly observation and self-check budget, not only code skill.
The depth axis is a ready curriculum. The paper flags the traces as future distillation and RL data. The benchmark is still being prepared for public release.
Agents do not see screenshots, so visual fidelity is out of scope. Apps are self-contained web systems, not external services or desktop/mobile. Construction is locked to GPT-5.6 Sol, so which behaviors get mined can move with the builder. Selector and harness changes could move both coverage and rankings. Reference-guided repair can be abused to clone proprietary products; the experiments stay on public repos.
Is the depth drop just longer context? An appendix holds step-level prompt size roughly fixed and still sees the drop. Gold patches pass, so the tasks are solvable. Agents lose on composition and observation, not on broken items.