Replication finds agent experience distillation preserves 44.1% of ICL gains on SWE tasks
burny_tech · x · 2026-07-29
A small-scale replication of Sample-Efficient Learning from Agent Experience reports that Experience-Conditioned Policy Distillation can preserve a meaningful share of in-context learning gains for coding agents.
- The team used an ensemble of GPT-5.6-Sol and Fable 5 research agents, then post-trained Qwen3.5-9B with Tinker on SWE-smith tasks.
- Their replication found EPD captured 44.1% of the ICL gains, while vanilla SFT captured 0%.
- Two ablations stood out:
- SFT on only the final successful trial underperformed ordinary SFT on all trials.
- Using four sampled teacher sequences per trial performed worse than using one, even when optimization steps were matched or increased proportionally.
- The chart also shows Fresh ICL as the strongest baseline, while EPD variants land between SFT and ICL on pass rate and normalized gain.
More from coding & agent
- 'LLM code is sloppy' rebutted with real ffmpeg code: identical if/else branches in a top library — kingslyj · 2026-10-16
- Office-editing tool calls need file, worktree and unit: lessons from dsh-univer-office — FunDistribution6537 · 2026-10-16
- Architect MD workflow: Gemini one-shot outputs that beat Opus, via 5-lens plan critique — oikk01 · 2026-10-16
- Prove a coding agent actually ran the tests: check artifacts, not approvals — ComprehensiveShake76 · 2026-10-16
- Building an acceptance harness for AI spreadsheet edits: allowlists, recalc checks, idempotency — ukdicha-modakkk · 2026-10-16
- Agent actions need checks outside the model: NVIDIA demos lean on guardrails amid wave of real agent incidents — Creamy-And-Crowded · 2026-10-16