ActionOPD distills agents at action level, lifting Qwen3 3.3-6.9 points on OOD benchmarks

nics-efc · hf · 2026-10-15

Problem: distilling LLM agents into smaller models suffers state-distribution mismatch with offline imitation, while token-level on-policy distillation (OPD) needs teacher token probabilities and compatible tokenizers.
Method: ActionOPD performs action-level on-policy distillation — at student-encountered states, actions are sampled from both student and a black-box teacher, and student actions are rewarded by semantic agreement, with no need for teacher token probabilities or reproducing teacher reasoning.
Results: training Qwen3-4B/8B with two teachers on curated tool-use tasks yields 3.3-6.9pp average gains across four OOD benchmarks; competitive with offline distillation and token-level OPD on τ²-bench; scales with student size and teacher capability across model families.
Extension: Self-ActionOPD with a same-model teacher and privileged context is also explored.

More from Research