Microsoft Paper: Distilling 50 Trajectories Into Skill Cards Matches Costly Reasoning
rohanpaul_ai · x · 2026-09-07
A new Microsoft paper explores paying reasoning cost once and reusing it: replacing expensive test-time reasoning with a small set of rules learned from past agent runs.
Method
- Collect 35–50 past trajectories, have a coding agent extract recurring failure patterns
- Turn them into a small markdown skill added to the non-reasoning model's system prompt
Results
- For GPT-5.4-mini, skills recovered 55%–100%+ of the gap between non-reasoning and reasoning modes across 4 agent benchmarks, using 2.9–4.5× fewer output tokens
- On ALFWorld and τ²-retail, the skilled non-reasoning model beat reasoning mode
- The distiller needed no reasoning traces: skills built from cheap non-reasoning rollouts were competitive in all 4 domains
Limits
- Reasoning still won on telecom and SpreadsheetBench, where instance-specific dependencies exceed what a fixed skill can capture
Takeaway: distill repeated procedures once; reserve test-time reasoning for tasks that genuinely need fresh search.
Related event: Microsoft Paper: Distilled Skill Cards Match Test-Time Reasoning(2 posts)→
More from coding & agent
- Dev demos AI models handling 3D objects and animations surprisingly well — chongdashu · 2026-10-23
- Jason Kneen burns 104.56B tokens and $24.7K in 304 days of AI coding on Codex and Claude Code — jasonkneen · 2026-10-23
- Typosquatted Blender MCP Repo Hit 958 Stars in Days and Silently Ran Malware — Promptmethus · 2026-10-23
- Benchmarking 56 fresh dependency bugs: MCP routing lifts Claude Code accuracy 68% to 91% — Available_Side_7134 · 2026-10-23
- Swapping only the eval harness moves results 30%: designing agent evals like a scientist — AI Engineer · 2026-10-23
- Med student builds a whole-library RAG with Cognee; single-book ingestion still freezes his machine — shrike293103 · 2026-10-23