Multi-Teacher Debate Distillation Breaks Single-Teacher Limits
青稞AI · wechat · 2026-07-13
This article introduces a paper from HUST: MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate. The core idea is to introduce multi-teacher debates into on-policy distillation (OPD) to break the capability limits of single-teacher distillation.
Methodology Highlights
- Traditional OPD allows the student to generate along its own trajectory, supervised token-by-token by the teacher. While closer to the student's distribution, it inherits the single teacher's blind spots.
- MAD-OPD has multiple teachers debate and correct each other based on the student's current state, distilling the post-debate consensus into the student. The debate logs are only visible to the teachers; the student only receives the final supervision signal.
- For Agentic tasks, the authors recommend using JSD as the divergence metric because it's more stable for long trajectories and doesn't depend on trajectory length or distribution overlap. For code generation, Reverse KL is preferred to ensure consistency in program paths.
Results
- In a Qwen 14B+8B-4B setup, MAD-OPD achieved AgenticAvg 25.69, CodeAvg 48.12, and OverallAvg 34.66, outperforming OPD and MT-OPD.
- On LiveCodeBench v6, the 4B student reached a pass@1 of 29.83, surpassing the 14B teacher's 25.57; under BoN@16, the 4B also hit 43.43, higher than the teacher's 33.14.
- Ablations show that simply averaging multiple teachers isn't enough; adding debates improves results, and incorporating confidence weighting works best. Two rounds of debate yield the best performance, while three rounds degrade due to context bloat and noise.
The article concludes by emphasizing that the goal of MAD-OPD isn't to make a small model magically surpass its teachers, but to organize the scattered, localized correctness of the teacher pool into a stronger training signal, thereby breaking the single-teacher ceiling.
More from coding & agent
- An MCP server signs every AI agent tool call into a verifiable Merkle chain — Funky_Chicken_22 · 2026-07-22
- Annotated transcript of a Claude Code team interview is now available — trq212 · 2026-07-22
- Claude Code skill uses 10 Markdown rules to make outputs ADHD-friendly — alex_verem · 2026-07-22
- BUZZ launches as an open-source group chat layer for teams and agents — Scobleizer · 2026-07-22
- A Firecracker-based platform says it can host 6,000 AI agents on one 256 GB server — maritime_sh · 2026-07-22
- A better path to agent autonomy is running waves, finding friction, and iterating — JnBrymn · 2026-07-22