DART-SD: Diamond-topology Aware Retrieval and Tuning for Self-Distillation of Multi-Turn Tool-Calling Agents
Hangrui Xu, Jiarui Wang, Yang Yang, Chuanbo Zhu, Fangda Chen, Ziqi Wu, Jingming Cai, Yan Song
cs.CL, cs.AI, cs.LG, cs.MA
2026-08-19
DART-SD distills a tool agent only after its first exit from the success graph. Qwen3-8B averages 45.58 on five benchmarks versus 41.64 for full-trajectory SFT.
In multi-turn tool use, many subgoals commute: weather then flights, or the other way around, can reach the same information state. Cloning one linear demonstration flattens that diamond lattice into a single path. Full-trajectory SFT overwrites exploration the student already got right. Outcome-level GRPO spreads a terminal score across every call and punishes good intermediate steps inside a failed rollout. Hindsight distillation still treats interaction as a rigid sequence, so a fatal error and a harmless detour look the same.
Compressing a large teacher into a small tool model needs a structure that can say "several legal paths meet in the middle." A longer demonstration does not provide that.
DART-SD builds an Interaction-State Transition Graph (ISTG) from teacher rollouts. Nodes are not raw actions. They are cumulative interaction states (It, Ut): I is the set of information atoms parsed from tool returns, U is the multiset of useless calls since the last informative step. A main node acquired a new fact; an auxiliary node spun its wheels. Paths that gather the same facts in different orders reconverge on a main node, which is where the diamond appears. Empty errors and non-informative payloads map to the empty set, so a failed response does not mint a fake state and blow up the graph.
Failed student traces are replayed under the same update and projected onto the teacher's success-reachable region: nodes that sit on a successful rollout and are still within a remaining-distance budget of a success terminal. The Critical Topological Breakpoint (CTB) is the first step from a projectable state to a non-projectable one. The student prefix before the CTB is kept. Teacher success and failure traces are retrieved as privileged context, and a recovery suffix is generated from the anchor. The loss hits only assistant tokens after the CTB and before the final answer. The prefix, user turns, tool observations, and the answer itself are masked, so mastered behavior is not rewritten.
Five rounds of this loop move the CTB later as the student improves. Teachers are a mix of Qwen3.6-27B and GLM-5.2. Students are Qwen3-4B and Qwen3-8B, trained on 2,215 verifiable FTRL tasks, eight rollouts per task per round, learning rate 5×10^{-7}. The main table is no-thinking.
On Qwen3-8B the five-benchmark average is 29.60 for the base model, 41.64 for full SFT, 40.33 for FTRL-GRPO, and 45.58 for DART-SD. In-domain FTRL Solve-F1 is 45.66 versus 41.89 for SFT. The 4B average is 39.17, still above SFT at 37.62. The paper reports that the 8B student beats the teacher on FTRL, ToolHop, and τ-bench with far fewer parameters.
The sweep is not uniform. On 8B BFCL Multi-Turn, FTRL-GRPO scores 35.25 and DART-SD scores 27.63. On 4B the same column flips: 23.88 versus 13.50. An average win is not a win on every board.
Mean tool calls on successful traces fall from 4.23 in round 1 to 3.55 in round 5, below the golden construction traces at 4.02. CTB position on failed traces moves from 0.348 to 1.452, so longer prefixes stay inside the recoverable region before the first break. Ablation on 8B FTRL Solve-F1: self-distillation 38.10 from a 23.48 base, CTB-local loss 39.51, progressive rounds 43.93, ISTG projection 45.66. IFEval rises from 34.75 to 45.29 and MMLU from 70.94 to 74.27, so tool tuning did not punch a hole in general skills.
Agent distillation should not paint loss on an entire trace. Commuting subgoals are a graph, not a sentence. Supervising only the first departure from the success region keeps alternative explorations and stops the optimizer from revisiting a correct prefix. For anyone shrinking a 27B-class teacher into a 4B or 8B tool model, that is a better next try than another full-trajectory SFT pass. Shorter successful traces and later breakpoints say the student found cheaper feasible paths, not a memorized teacher script.
MatchTIR still assigns turn-level rewards on linear traces. DART-SD drops the linearity assumption. The cost is that you need a teacher graph first.
There is no standalone limitations section. The ISTG depends on successful teacher traces and a semantic atom map. The authors say labeling errors tend to omit atoms rather than invent them; omission still thins the graph and tightens projection. The diamond story is for order-independent subgoals. On strictly chained tasks the collapse is milder and the gain should shrink. The 8B BFCL loss to GRPO is large enough that the average cannot cover it. Training lives inside FTRL's verifiable environments; dirty API payloads, auth failures, and user interruptions are not equally stressed. Two teachers are mixed with no single-teacher control. Atom assignment looks at a task's candidates jointly, so graph quality is glued to that step, and labeling error is never measured on its own.