TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning
Changle Qu, Sunhao Dai, Hengyi Cai, Yuqi Zhou, Xinran Chen, Simon, Jun Xu
cs.CL, cs.AI
2026-08-05
TurnSight does turn-level hindsight self-distillation: it builds teacher views from the agent's own real trajectory at several lookahead horizons, votes on a reliable direction, and modulates RL advantage magnitude without flipping its sign. Qwen3-8B reaches 42.02 overall across three benchmarks, 7.7% over the best baseline.
When LLMs solve problems with tools (Tool-Integrated Reasoning, TIR), reinforcement learning hits a stubborn wall: credit assignment. By the end of a trajectory you know whether it succeeded, but not which intermediate tool calls helped, which were redundant, and which hurt. Outcome-based RL such as GRPO propagates a single trajectory-level advantage to every action, treating decisions with very different effects as equal.
Existing self-distillation tries to densify the signal, but its teacher's privileged context usually comes from ground-truth answers or reference rollouts that do not match the states the agent actually visits. And token-level supervision misses the structure of a tool interaction: it can hand out conflicting credit within a single turn. TurnSight attacks both. The signal should be dense and rooted in the agent's real path, and it should be scored per turn, not per token.
TurnSight is turn-level hindsight self-distillation, and the design breaks into a few moves:
In one line: it builds supervision from the agent's real trajectory, scores it per turn, and adjusts magnitude without changing direction.
Trained on Qwen3-4B and Qwen3-8B (FTRL dataset, 2,215 problems; MatchTIR as the RL backbone), evaluated on three benchmarks: FTRL (in-domain), BFCL and ToolHop (out-of-domain).
| Method (8B) | FTRL avg | BFCL avg | ToolHop | Overall |
| MatchTIR (best baseline) | 42.78 | 33.78 | 40.54 | 39.03 |
| TurnSight | 46.92 | 37.56 | 41.58 | 42.02 |
On 8B, overall average climbs from 39.03 to 42.02 (about +3 points absolute, roughly +7.7% relative), with out-of-domain gains too, so it is not overfitting the training distribution. On 4B, overall goes from 34.76 to 37.51.
Ablations (8B, FTRL avg): dropping turn-level aggregation falls to 43.23, group normalization to 43.65, multi-teacher selection to 45.62. All three pull weight. Two notable points: using only tool returns (no ground-truth answer) works best, and among fixed lookaheads one step ahead is strongest while longer horizons hurt. Covering the full trajectory beats first-five-turns (44.56) and last-five-turns (43.04).
Tool use is table stakes for agents, yet most open RL training is still stuck at outcome-level rewards. TurnSight offers a denser signal that needs no ground-truth answer and uses only the agent's own experience, and it plugs into existing GRPO and MatchTIR pipelines (code is open). It is an incremental but real gain: credit moves from trajectory level to turn level, at the cost of extra teacher rollouts with privileged context. If you train agents, it is worth trying on your own long-horizon tool tasks.