Reason Wide, Not Deep: Amortizing the Reasoning Premium into Distilled Skills
Agamdeep Singh, Srishti Gautam, Priyanshu Gupta, Nikita Mehrotra, Tanmay Bakshi, Sumit Gulwani
COLM 2026 Efficient Reasonin
cs.AI
2026-08-08
Distilling procedures from a few dozen trajectories into a skill appended to the prompt recovers 55-100%+ of the reasoning gap on four agent benchmarks at one-sixth the tokens.
Reasoning models beat non-reasoning ones on multi-step agentic tasks, but they pay for it with 2.7-6x more output tokens per episode. A lot of those tokens are spent re-deriving the same procedures, the same "check the email, then call the API" flow, on every single run. That cost is linear in calls: the more you run, the more you pay to re-think what you already knew.
The Microsoft team (Sumit Gulwani's group) makes a simple observation: within a domain, those repeatedly derived procedures are shared knowledge that does not change from task to task. If so, extract it once and reuse it, instead of asking the model to reconstruct it on every deployment.
They call it passive skill distillation, in three steps:
The point is that it distills procedural knowledge, not parameters: a natural-language rule that a human can read, edit, and move to any model whose prompt you can touch.
On GPT-5.4-mini across four agentic benchmarks (success rate / mean output tokens):
| Benchmark | Reasoning | Non-reasoning | Non-reasoning + skill |
| ALFWorld | 0.713 / 3723 | 0.567 / 952 | 0.787 / 832 |
| SpreadsheetBench-Verified | 0.613 / 3291 | 0.447 / 960 | 0.560 / 831 |
| tau2-telecom | 0.450 / 2143 | 0.192 / 421 | 0.333 / 597 |
| tau2-retail | 0.350 / 1615 | 0.325 / 536 | 0.408 / 565 |
On ALFWorld and retail, non-reasoning-plus-skill outright beats reasoning mode, at 3-4.5x fewer tokens. Across the board, skills recover 55%-100%+ of the reasoning gap while spending zero reasoning tokens.
On Qwen3.6-27B the gap is wider still: ALFWorld reasoning 0.773 vs. non-reasoning+skill 0.980 at 14.9x fewer tokens, and on telecom the skill matches reasoning (0.933) at 5.9x fewer tokens.
Two more comparisons matter. First, reasoning trajectories are not a prerequisite: skills distilled from non-reasoning trajectories alone stay competitive with skills from a paired corpus. Second, against GEPA, an automatic prompt-optimization method, the distilled skill wins on retail (0.458 vs. 0.392) at 4.1x lower production cost.
For anyone deploying agents, this is a real cost lever: a one-time distillation of a few dollars buys a 2.7-6x token reduction on every call, and in the best cases higher accuracy too. The skill is plain text, touches no weights, and drops into any model whose system prompt you can edit. Adoption cost is low.
It also punctures a common assumption: much of the gain from reasoning mode comes from knowledge that could have been made static. If it can be made static, there is no reason to recompute it on every episode.
The authors list several. Skills are distilled once per model-domain pair, with no test of cross-model transfer. The study covers two models and four domains. They measure evaluation variance (3 seeds) but not distillation variance, so a fresh run of Claude Code might produce a different skill.
More importantly, on telecom and SpreadsheetBench the skill falls well short of reasoning mode. That is the per-instance knowledge distillation cannot capture, where genuine step-by-step reasoning does irreplaceable work. Qwen also regresses on retail, so the process is not uniformly reliable. And because skills are natural language, whether the model actually follows them or gets overridden by other instructions is not systematically tested.