Teacher steering recovers quantized LLM agents to within 1 point of BF16 with 67% less training

SteerDistill: Quantizing LLM Agents via Closed-Loop Teacher Steering

Genming Cui, Xiangyu Li, Zixu Hao, Xin Ding, Yunxin Liu, Ting Cao

cs.AI, cs.LG

2026-11-10

A BF16 teacher steers a quantized agent through failed turns, then distills the fixes; it lands within 1 point of BF16 across five agent benchmarks at 67% less training cost.

What problem this solves

Agents burn tokens: one task means many model calls, so low-bit quantization is the standard cost fix. The catch is that quantized agents fail in a characteristic way. They overlook task requirements or environmental conditions and drift off the full-precision trajectory: calling cat on a file before entering the workspace, when the tool only reads the current directory, or creating three requested files and leaving two of them empty. The AIR, Tsinghua team calls this behavioral divergence, a mismatch in environment transitions rather than garbled text.

Fixing it hits a data deadlock. Offline distillation corpora were collected without this student, so they rarely cover the contexts where it diverges. On-policy rollouts do reach those contexts, but the student almost never produces a correct action there: in a BFCL V4 diagnostic, four unguided retries from the same pre-error state yield at least one correct action only 9.6% of the time (turn-level Pass@4).

Method

Each training cycle has two stages: the teacher steers in a closed loop, the student revises itself, and the fix gets distilled.

Constructing corrections. A failed rollout is aligned against a successful teacher reference for the same task and initial state, matching on environment transitions only — different actions that produce the same transition count as equivalent. At the first divergent turn, history and environment are rolled back to the pre-error state. The teacher writes a structured steering prompt stating the discrepancy, the conditions under which the corrective rule applies, and the fix. The frozen student snapshot regenerates and executes under that prompt, and the result is checked against the reference transition (local verification). On failure the teacher refines the prompt using execution feedback and the student retries, up to four attempts per turn. Verified corrections are kept even when the task ultimately fails, and the student continues unguided afterward.

Internalization. Training never shows the student the prompts. At corrected turns it matches the steered snapshot's token predictions (conditioned on context plus prompt) with forward KL; everywhere else it matches the full-precision teacher with reverse KL. Inference runs the student alone. Steering lifts turn-level Pass@4 at divergent turns from 9.6% to 82.3%.

The ablations justify each choice. Intervening at behavioral divergence beats intervening at action mismatch by 1.83 points, since different actions can be equivalent; guided self-revision beats unguided retry by 2.38 and teacher takeover by 1.78; swapping the prompt source from the BF16 teacher to the student itself costs 5.10 points, and to GPT-5.6-Sol costs 1.57.

Results

Five agentic benchmarks (BFCL V4, τ²-Bench, WebShop, ALFWorld, Search QA), three models, teacher and student from the same checkpoint. Qwen3-8B under W4A16:

MethodFive-benchmark avg
BF16 reference53.54
QeRL (strongest quantized baseline)51.00
SteerDistill53.18

That is 0.36 points from BF16. On Qwen3.5-4B and Gemma 4 12B-it it leads QeRL by 1.71 and 1.67 points (60.95 and 64.31 average), each within one point of BF16. At W4A8 the margin over QeRL is 1.93 points; at W2A16 it is 3.39 over On-Policy QAD, with one miss: BitDistiller edges it 34.22 to 34.05 on Search QA. Training efficiency on τ²-Bench: 32 GPU hours reach 31.68% success versus QeRL's 29.84% at 96 GPU hours, 67% less compute. Inference is cheaper too, at 4.0K output tokens per successful task against 4.1K–5.2K for the quantized baselines (BF16: 3.9K), with fewer interaction steps and tool calls.

Why it matters

Teams deploying quantized agents get a near-lossless recipe at W4A16 and one that no longer collapses at W2A16, with code released. Compared with verifiable-reward RL like QeRL, the supervision is a dense, locally verified correction rather than sparse task success, which buys two-thirds less training compute and shorter successful trajectories. The teacher is just the BF16 twin of the same checkpoint; even GPT-5.6-Sol as the prompt source does worse.

Limitations

Stated by the authors: the pipeline needs successful teacher references (no corrections where the teacher itself fails), needs the environment rolled back to pre-error states, and needs reliable local verification — settings without state restoration or verifiable transitions are untested. Local verification only checks the effect at the current turn; it guarantees neither final task success nor correct reasoning. Beyond the self-report: all three models sit in the 4B–12B range with teacher and student from the same checkpoint, so the strong-teacher-weak-student setup is unverified; every benchmark is a text environment with precisely matchable state, and transfer to messy settings like real web pages is an open question; the training loop, with repeated prompt generation, retries, and environment rollbacks, carries real engineering weight.

Terms

Source

What people are saying

Related papers

All paper explainers