Socratic Skill Learning: Interleaving Teaching and Practice for Transferable Agent Skills
Letian Zhu, Han Li, Yifan Yao, Xinping Lei, Xinyu Che, Jiahao Wang, Dianyi Wang, Yucheng Fu, Jiakai Wang, Zhiqi Bai, Wenbo Su, Jiaheng Liu
cs.AI
2026-10-13
Student-written skills with interleaved teaching score 0.801 utility on 155 held-out cases, vs 0.723 with none. Issue-guided review hits 0.805 and cuts educator tokens by 61.7%.
An Agent Skill is a reusable instruction sheet plus optional scripts. It travels with an LLM agent and does not change weights. A run that succeeds often never checks when the procedure holds, or which allowed variations should still use it. Write that untested assumption into the package, and the next session will execute it as a rule.
Trace2Skill extracts local lessons from trajectories after the attempt. SKILL-KD proposes patches from teacher-student trace contrasts and keeps a patch only if a student rerun passes. SkillLearnBench delivers teacher feedback after a failure. What gets revised depends on which hypotheses the learner chose to test. A comment while evidence is still being collected changes that set. A review after the draft is finished does not.
SkillTransferBench has 35 task families and 408 executable cases, adapted from Terminal-Bench, AgentIF-OneDay, SpreadsheetBench, CodeContests, FullStackBench, and OfficeBench. Each family publishes an invariant goal and the axes that may change. Practice uses one anchor case. Held-out cases are solved in fresh sessions that load only the frozen package. The main cohort keeps 22 families and 155 held-out cases: at least five verified held-out cases, and at least two non-perfect No-Skill outcomes in screening.
Socratic Skill Learning (SSL), from Nanjing University and Alibaba, gives the environment and the package to the Student and the comments to an Educator. The Student runs the target solver model. The Educator may read recent reasoning, tool results, and the live package, then explain, challenge, diagnose, or stay silent. The Student may question, rewrite, or reject the advice. Only the package leaves the session: SKILL.md plus optional scripts and references.
A fresh solver is a new session that loads the frozen package and none of the acquisition chat. At the end of each round it reruns the reset anchor, and that score enters the next round. Student and Educator sessions persist. Exam sessions start clean. Every condition runs three rounds.
On the main runs the Student and fresh solver are Qwen3.7 Plus and the Educator is Qwen3.7 Max. DeepSeek-V4-Flash with Pro, and Kimi K2.5 with Kimi K3, each acquire their own packages. Comparators are No Skill, Expert Authoring, Post-hoc Teaching (one review after the trajectory each round), and Trace2Skill and SKILL-KD adapted to these families.
The score is family-macro Avg@3: equal weight per family, with cases and repeats averaged inside the family. Qwen acquires three times per condition. Timeouts score zero.
| Method | Utility | vs No Skill | Full pass | Acquire tokens/run |
| No Skill | 0.723 | baseline | 32.5% | none |
| Expert Authoring | 0.728 | +0.46 pp | 40.4% | 4.95M |
| Post-hoc Teaching | 0.748 | +2.44 pp | 40.4% | 8.71M |
| Trace2Skill (adapted) | 0.761 | +3.76 pp | 39.1% | not on this accounting |
| SKILL-KD (adapted) | 0.751 | +2.76 pp | 41.7% | different scope |
| Fixed-Window SSL | 0.801 | +7.75 pp | 49.5% | 6.89M |
| Issue-Guided SSL | 0.805 | +8.18 pp | 50.8% | 5.72M |
The family-resampled interval for Fixed-Window over No Skill is [3.53, 12.68] percentage points. Expert Authoring and Post-hoc Teaching both have intervals that cross zero. Fixed-Window leads adapted Trace2Skill by 3.99 points, adapted SKILL-KD by 4.99, and Post-hoc Teaching by 5.31, with 21% fewer acquisition tokens. Full pass, every private check succeeding, moves from 32.5% to 49.5% and 50.8%.
On this Qwen cohort both SSL schedules take the top two spots in every domain. Of 155 cases, Fixed-Window improves 72, leaves 59 unchanged, and regresses 24. It beats No Skill in 16 of 22 families. Flask rises from 0.408 to 0.785 under issue-guided review. On bank-transaction filtering every learned package scores below the No-Skill mark of 0.678.
Gains on the shared lite subset, 10 families and 77 cases, in percentage points over each pair's own No-Skill solver:
| Protocol | Qwen | DeepSeek | Kimi |
| Fixed-Window SSL | +4.1 | +3.1 | +8.2 |
| Issue-Guided SSL | +4.5 | +2.5 | +7.4 |
| Expert Authoring | +1.1 | +1.2 | +5.8 |
| Post-hoc Teaching | -0.7 | +0.5 | +4.1 |
Qwen averages three repeats. DeepSeek and Kimi use one. Interleaving ranks above expert writing on all three pairs. DeepSeek's Fixed-Window interval versus No Skill is [-4.63, +13.03] and includes zero. Kimi reaches 0.762 against 0.679 with no skill.
Across 630 reviews, the share that delivers a lesson falls from 73.0% in round 1 to 16.1% in round 3. After a perfect anchor, the package still changes in 98 of the next 99 rounds, 66 of them with no new lesson. Relative to silence, a lesson raises the chance of a file write on the next call by 23.9 points, and of a package change within ten calls by 38.6 points. Review timing follows the Student's current work, so the gap is an association.
In one HTML-finance run the Educator tells the Student to recompute figures from other lines. Generic arithmetic flags four metrics and the wrong line. The bad gross margin is line 210: the report says 60.8%, while 520.4/842.0 times 100 rounds to about 61.8%. Financial relations isolate that line. A fresh solver then passes 4/4 on a held-out cash-flow statement.
Tokens per review grow from 74.7K to 395.7K before follow-up dialogue, about 5.3 times, and silence accounts for 85.8% of final-round review tokens. Issue-guided review reaches 0.805 utility, cutting Educator tokens from 2.40M to 0.92M per run (61.7%) and total acquisition tokens from 6.89M to 5.72M (16.9%). Gain per Educator token is 2.75 times the g=10 window. Review count falls 47.8%, and tokens per review fall 26.7%. Calling the Educator every 5 or every 20 tool calls both land at 0.776 utility. DeepSeek and Kimi cut Educator tokens by 63.8% and 77.4%, staying within one point of Fixed-Window. Total tokens rise for DeepSeek, where the Student does more work, and fall for Kimi.
A skill package is the usual way to carry a procedure into a new session without a weight update. Letting the stronger model write that package beats No Skill by 0.46 points on the main cohort, with a wide interval. Keeping the Student as author and interrupting during practice beats a review that arrives after the draft, by 5.31 points and with fewer acquisition tokens. The lesson says what to check. The runs decide what is safe to keep.
Issue-guided review is the schedule you would leave on: 0.805 utility and 61.7% less educator spend, if the task can be stated as an anchor, a public variation spec, and an automatic checker. The lead over trajectory distillation is about four points, lower bound 1.45. That is an incremental gain. Office families where No Skill already scores 0.904 have little room left.
The appendix states the limit in one sentence. The study compares complete protocols on a shared single anchor, and does not separate the Educator, the review schedule, and the issue tree.
Variation axes are visible during practice. Held-out cases change values on those axes. They are not task types outside the spec. The headline numbers are Qwen3.7 Plus with Max. DeepSeek and Kimi use 10 families and one repeat, and the DeepSeek gain interval includes zero. Both distillation baselines are adapted. SKILL-KD's 128.11M tokens over 66 runs use a different cost scope, so that bill is not comparable.
Fixed-Window regresses 24 cases. Among 25 cases that No Skill already solves perfectly, it breaks 3 and Issue-Guided breaks 5. Repeat utilities are 0.772, 0.799, and 0.831, standard deviation 0.030, against 0.006 for No Skill. The Educator is always the stronger model in the pair. Packages keep changing after a perfect anchor, so extra search and extra teaching stay tangled.