Stabilizing On-Policy Targeted Guidance in Agentic Reinforcement Learning
Kevin Frans, Alexander M Rush, Sergey Levine, Pieter Abbeel, Ashvin Nair
cs.LG
2026-11-04
On Qwen3-Coder-30B agents, failing-turn self-distillation plus a per-criterion RL reward lifts Markdown from 0 to 0.85 while coding pass rate stays at 0.33, matching GRPO.
In a multi-turn coding agent, a tuned GRPO run already moves pass rate. Rewards outside the base policy's support do not. A tool argument missing from supervised finetuning, or a writing rule the model almost never samples, gives policy gradient no positive examples, so the score stays at 0.
The judge usually already wrote the missing instruction. Standard RL drops that sentence and keeps a scalar. Targeted guidance puts the sentence back in front of the failing turn and distills the same weights on-policy, token by token. The base model is Qwen3-Coder-30B after agent supervised finetuning. Pass rate is meant to hold. The new target is whatever naive RL cannot enter.
Each binary criterion comes with a fixed feedback sentence. On a failed turn the sentence is inserted immediately before that turn. The expert is the live policy with this extra context. The update stops the gradient through the expert and matches the student to it with reverse KL on that turn's tokens. Other turns are masked.
The loss is GRPO plus distillation on the failed-turn set only, weighted by λ. KL is computed on the trainer over the full vocabulary, so the target is not a sampled estimate that drifts from the inference policy. Tokens in a rollout are normalized together, which keeps looped, very long traces from dominating the gradient.
Feedback stays out of the global prompt and touches only the bad turn, so student and expert differ in a small place. Once the turn passes, it leaves the failed set and distillation shuts off. The per-criterion RL reward stays. Distillation moves the new behavior into the support. The policy gradient is what retains it. Distillation without that reward learns the behavior and then loses it.
A feedback sentence is kept only if the base model scores higher with it than without it. During training the expert does not rewrite the turn. The extra cost is one forward pass for logits. After every criterion holds consistently, distillation is zero and the update is ordinary policy gradient. Gating on failures is what stops a spurious KL from surviving at an optimum. Distilling every turn does not have that property.
Training runs 200 steps with 32 groups of 32 rollouts, Adam, and learning rate 3e-6. Each software-engineering prompt is seen once. The sampler may lag the parameters being updated. Thinking tokens and tool-call tokens are both in the KL. λ is swept, and the chosen value is not reported.
| Method | Pass rate | Markdown | Approval |
| Baseline (coding only) | 0.33±0.05 | 0.00 | 0.00 |
| GRPO (style as reward) | 0.33±0.07 | 0.00 | 0.00 |
| OPSD (distill only) | 0.30±0.05 | 0.77±0.04 | 0.25±0.09 |
| Online SFT | 0.26±0.05 | 0.01±0.01 | 0.02±0.01 |
| LaTRO | 0.14±0.09 | 0.00 | 0.00 |
| Verifree | 0.20±0.06 | 0.01±0.02 | 0.01±0.01 |
| Targeted guidance | 0.33±0.05 | 0.85±0.09 | 0.55±0.06 |
On Markdown and Approval, baseline and GRPO stay at 0.00. Targeted guidance reaches 0.85±0.09 and 0.55±0.06, with pass rate still 0.33±0.05. OPSD hits 0.77±0.04 on Markdown but only 0.25±0.09 on Approval, and pass rate slips to 0.30±0.05. Online SFT, Verifree, and LaTRO fall to 0.26±0.05, 0.20±0.06, and 0.14±0.09. All three sample a new turn from the feedback-conditioned expert. Those turns can satisfy style while raising actions that hurt coding, and under asynchronous sampling they go stale.
Criteria already in support do not need this trick. One sits at 0.70±0.06 for the baseline, 0.96±0.02 for GRPO, and 0.95±0.01 for targeted guidance. Another moves from a baseline of 0.04±0.03 to 0.48±0.08 with GRPO, 0.29±0.03 with OPSD, and 0.77±0.05 with targeted guidance. A third stays at 0.00 for GRPO and 0.01 for OPSD, and reaches 0.64±0.13 only with targeted guidance.
A new semantic-search field, how many lines to return, is ignored by the base model even when the prompt shows an example. Targeted guidance learns the format without a coding regression. The loss sits on the delimiter tokens and fades once the format is fixed. When every turn must include a status poem, GRPO still lifts pass rate and never moves the poem reward. Distillation without the poem reward, or feedback on every turn whether it passed or not, produces poems and a clear pass-rate drop. Targeted guidance picks up the poem with a smaller regression. Neither setting prints an exact pass rate.
Too large a λ spikes entropy, lengthens outputs, and then drops both pass rate and style. A small λ is stable and slower on style. Leaving thinking tokens out of the KL makes style criteria rise and fall. With thinking included, the curves rise together, and the trace starts reminding itself to use markdown and to check with the user. Without that pressure, the closing trace talks only about the code.
The work is from Cursor and UC Berkeley, and it sits beside an existing GRPO loop. The added cost is one forward pass on failed turns, plus a hand-written feedback line. The fit is a new tool field, a format supervised finetuning never covered, or a rubric item that stays at zero because it is never sampled.
Pass rate does not beat coding-only GRPO. Format and style can leave zero. Nothing here shows stronger bug fixing. Against Online SFT, LaTRO, and Verifree, which let the expert rewrite a turn, the student stays on its own rollouts and gives up less of the main task.
Composer 2.5 used a version of this technique. That release is not a Qwen model. Every number above is from public Qwen3-Coder-30B.
Every criterion needs a hand-written sentence, then a filter on the base model. Whether a judge can write hints that do not fight the main reward is untested.
The only environment is multi-turn software engineering, and the targets are mostly style and format. A poem shows that an out-of-support behavior can be taught. It does not show the same for bug fixing or long-horizon planning. Reverse KL, forward KL, Jensen-Shannon, and Rao-Blackwellized KL behaved similarly, without a careful comparison. Earlier OPSD variants were not run head to head in this setup.
Entropy spikes and style oscillations were held down by a smaller λ and by distilling thinking tokens. That is one coding stack. The pass-rate drops for LaTRO and Verifree are attributed to stale expert actions under asynchronous sampling. There is no synchronous ablation, so that story is still a hypothesis.
The expert is assumed to beat the current policy and to stay close to it. Those are design heuristics, not something the runs establish. Feedback is filtered with autoregressive scores and then applied as per-token distillation. Those two checks are not the same.