Half of agent failures trace to one early mistake; PivotOPD teaches recovery, beats 13 baselines

PivotOPD: Learning to Recover from Pivotal Mistakes in Multi-Turn Agents

Yinghui He, Yapei Chang, Khushi Bhardwaj, Daniele Molinari, Tugrul Konuk, Jan Kautz, Ali Hatamizadeh

cs.AI

2026-10-01

PivotOPD adds teacher supervision at and after pivotal mistakes, training agents to avoid and recover from them; it beats 13 baselines on three agent benchmarks, +5.5% on ALFWorld.

What problem this solves

On-policy distillation (OPD) gives a student agent dense, token-level teacher supervision on trajectories the student generates itself. In multi-turn settings it has a structural blind spot: every action changes the environment, so one wrong action drags the agent into states it would never otherwise visit, and errors compound turn over turn.

ALFWorld, a text-based household environment, ships with a symbolic oracle that computes the shortest remaining path to the goal. Using it, the authors put numbers on the problem across Qwen3-8B, 30B, and 235B: 59% of failed trajectories contain a pivotal mistake, an action that lengthens the remaining optimal path or makes the task unsolvable. The first one arrives early, at a median of turns 8-12 out of 30, after which the models burn an average of 18-21 more turns without recovering.

The most useful finding is that most of these mistakes are still fixable. In counterfactual replays of Qwen3-8B's 72 failures containing a pivotal turn, swapping the pivotal action for the oracle action lifts success from 8% to 59%. Keeping the mistake and forcing oracle actions at just the next two turns still reaches 58%. Recovery is nearly as effective as prevention.

Standard OPD delivers neither. After 120 training steps, overall failure falls 23.6% while pivotal-turn failures fall only 2.1% (51% to 49%). OPD does crush the committed mistake, with median probability down from 0.999 to below 10^-5, but the correct action stays under 1% probability at every pivotal turn. A group of 8 rollouts is not expected to sample it once, so there is nothing to learn from.

Method

PivotOPD layers two distillation terms on top of group-based RL and concentrates supervision at pivotal turns and the few turns after them.

Pivot detection. A teacher LLM, larger than the student, reads each finished rollout along with its outcome, nominates up to m candidate turns, and names a gold action at each. A candidate counts as pivotal only when the student's committed action disagrees with the gold action. Detected turns land within one turn of the oracle-labeled one in 77.8% of failed ALFWorld rollouts on average, at least twice the rate of random selection. No oracle is used in training, so the pipeline is identical across benchmarks.
Preventive distillation. At each pivotal turn, a privileged self-teacher re-scores the response the student already produced. The self-teacher is the frozen student conditioned on a short hint naming the gold action, and a reverse KL loss shifts the student toward it. The teacher only names actions; the hinted student writes the text, so targets stay in the student's own style.
Recovery distillation. After a pivotal turn, the teacher names a recovery action at each of the next K turns. The hinted self-teacher writes recovery responses at those states, and the student trains on them without the hint under a forward KL loss. From the second recovery turn on, actions are executed in a replayed copy of the environment, so later contexts reflect the real post-mistake world.

The split between the two KLs is the core design move. Updates sampled from the student (outcome RL, reverse KL) have an expected gradient on an action proportional to the student's own probability p(a), so a recovery action with near-zero mass gets a vanishing update. Recovery distillation takes its expectation under the self-teacher, so the update size tracks q(a) and plants probability mass exactly on the modes the student has starved; Proposition 1 formalizes this. All three losses run inside a single PPO update.

Results

Main comparison: Qwen3-1.7B and 8B students on ALFWorld, WebShop, and Search-based QA against 13 baselines, first place on all eight per-benchmark averages.

Setting	Comparison	Margin
1.7B, ALFWorld avg	strongest baseline SDAR 68.2	73.7, +5.5
1.7B, Search-based QA	strongest baseline	+5.9
1.7B, WebShop success rate	RLSD	+14.1 (partial-credit score only +1.2)
8B, three benchmarks	strongest baseline	at least +1.8
Self-distillation, student as own teacher	strongest baseline	+3.9 avg, first on all three
Nemotron-3.5, SWE-Bench Verified	standard OPD 63.0 (+0.2)	66.0 (+3.2)

The mechanism checks line up. Replaying the 72 pivotal mistakes, PivotOPD recovers about nine times as often as the base model and 26.9 points more often than the preventive-only variant; at 8B it recovers more than three times as often as standard OPD. Dropping recovery distillation (K=0) costs 21.5 points of best WebShop validation score at 1.7B, so most of the gain really comes from recovery training. On WebShop the gains concentrate in the success rate, which requires fully completed purchases, rather than in the partial-credit score, consistent with pushing half-finished tasks to completion. Injecting the same hints at random turns collapses the ALFWorld average to 62.6 against 73.7 for the full method, so placement matters.

Without any stronger teacher, self-distilled PivotOPD still wins all three benchmarks (+3.9 on average), though ALFWorld sits 10.2 points below the stronger-teacher setting. Where and how the teacher intervenes matters more than raw teacher capacity.

Why it matters

The paper turns a vague intuition about multi-turn failure into a measurable structure: most failures trace to one early, largely recoverable mistake, and neither outcome RL nor standard OPD produces a learning signal at the states such mistakes create. Group-relative advantages are identically zero when every rollout of a task fails, and OPD never samples the recovery action. For anyone training agents with RL, this is a directly copyable recipe, and it pays off most where successful rollouts are scarce, which means small students and hard environments. The privileged self-teacher also keeps teacher cost low: the teacher names actions and never writes demonstrations.

Keep the effect size in proportion: at 8B the margins compress to the +1.8-2% range, a steady but modest gain. The large wins sit at 1.7B.

Limitations

The SWE-Bench +3.2% never exercises recovery distillation. Episodes there are long and containerized, so the authors audit only the final committed action (m=1), set K=0, run prevention only, and take the hinted distribution from the teacher itself. Whether the recovery mechanism transfers to software engineering is left to future work; this number is not evidence for it.
Pivot detection rests on teacher judgment: 71.1-84.4% agreement with the oracle within one turn, with known false positives. Free-text actions such as search queries rarely match verbatim and are treated as pivotal almost by default. The paper argues false positives are cheap but does not quantify their cost separately.
The diagnosis runs only on ALFWorld because it is the only benchmark with a symbolic oracle. Whether "over half of failures hinge on one pivotal mistake" holds on WebShop, QA, or SWE-Bench is extrapolation, not measurement.
Recovery turns require replaying the environment in copies, capped at 64 recovery sequences per step; the budget K and recovery weight are tuned per benchmark on validation (K=1 on WebShop and QA, K=2 on ALFWorld).
The 59% counterfactual success uses oracle actions; real training relies on weaker teacher-named actions.

Terms

On-policy distillation (OPD): token-level teacher supervision applied to trajectories the student generates itself.
Pivotal mistake: an action that lengthens the shortest remaining path to the goal or makes the task unsolvable; the turn it lands on is the pivotal turn.
Privileged self-teacher: the frozen student conditioned on a hint naming the correct action, used to produce teaching targets in the student's own style.
Reverse KL vs. forward KL: reverse KL takes expectation under the student and can only reweight what the student already produces; forward KL takes expectation under the teacher and raises the probability of actions the student rarely samples.
GRPO: group-based RL that rolls out one task several times and computes advantages relative to the group; identical returns give zero advantage.
Oracle action: the next action from a symbolic solver with full environment state, used for diagnosis only.

Source

What people are saying

Related papers

All paper explainers