Microsoft Paper: Distilling 50 Trajectories Into Skill Cards Matches Costly Reasoning
rohanpaul_ai · x · 2026-09-07
A new Microsoft paper explores paying reasoning cost once and reusing it: replacing expensive test-time reasoning with a small set of rules learned from past agent runs.
Method
- Collect 35–50 past trajectories, have a coding agent extract recurring failure patterns
- Turn them into a small markdown skill added to the non-reasoning model's system prompt
Results
- For GPT-5.4-mini, skills recovered 55%–100%+ of the gap between non-reasoning and reasoning modes across 4 agent benchmarks, using 2.9–4.5× fewer output tokens
- On ALFWorld and τ²-retail, the skilled non-reasoning model beat reasoning mode
- The distiller needed no reasoning traces: skills built from cheap non-reasoning rollouts were competitive in all 4 domains
Limits
- Reasoning still won on telecom and SpreadsheetBench, where instance-specific dependencies exceed what a fixed skill can capture
Takeaway: distill repeated procedures once; reserve test-time reasoning for tasks that genuinely need fresh search.
Related event: Microsoft Paper: Distilled Skill Cards Match Test-Time Reasoning(2 posts)→
More from coding & agent
- tenobrus: Human-centric code conventions may not survive superhuman coding models — ricklamers · 2026-09-07
- IFM ships K2 Horizon: 6 open-weight models you can self-host with vLLM or run locally via Ollama — HongyiWang10 · 2026-09-07
- Two 18-year-olds built a $240K ARR app business in 6 months with Rork — rudrank · 2026-09-07
- One prompt rebuilt a chess tactics site end-to-end with GPT-6 — BorisMPower · 2026-09-07
- Claude's prompt-audit command flags obsolete instructions before model upgrades — lydiahallie · 2026-09-07
- Agent records narrated demo videos of its own PRs using Playwright — dctanner · 2026-09-07