Ask to Distill: Learning to Synthesize Data for On-Policy Distillation
Jiashuo Sun, Hao Zhang, Xueqiang Xu, Heng Wang, Yixuan Xie, Pengcheng Jiang, Zhenrui Yue, Jiajun Fan, Peiran Li, Ge Liu, Shaowen Wang, Jiawei Han
cs.AI, cs.CL
2026-10-20
A2D scores questions by student-teacher disagreement and compatibility, with no answers. Three math pairs beat real-bank OPD; a 1.5B student recovers 99.9% of the gap.
On-policy distillation trains the student on trajectories the student samples, and the teacher supplies next-token distributions on those prefixes. The question bank can be tiny. Small subsets, sometimes a single question, recover most of the gain from a large bank. The question carries no reference answer. It steers this student into states where the gap with this teacher shows up.
A new question has no known correct answer, and updating the student once per candidate is too expensive to use as a scorer. Keeping the real questions with the largest teacher-student KL is a weak filter too. Disagreement can sit inside the student's top-K, where the teacher still has a correction the update can apply, or it can sit on tokens that update never touches. Candidate-restricted OPD writes advantages only on that top-K, so the second kind of disagreement has nowhere to land.
Ask-to-Distill (A2D) treats question writing as distribution elicitation. The synthesizer is a copy of the student, trained with GRPO: several questions are drawn for one instruction, and only the question policy moves, using group-relative advantages. The student and teacher used for scoring stay frozen. There are no external seed questions and no correctness reward. A frozen Qwen3-Embedding-4B model deduplicates the bank after training and does not enter the reward.
The frozen student answers each candidate once, inside a response budget. Disagreement δt is the teacher-to-student forward KL on the union of the two top-K supports. Compatibility ωt is the teacher's probability mass on the student's own top-K, which is the set the downstream update actually uses. The position score is st = min(δt/δref, 1) times ωt, so both factors have to be high at the same token. δref is a percentile of token-level disagreement on calibration questions the student samples itself, then held fixed.
The question reward sums st with exponent α = 0.5, then multiplies by a length penalty and a within-batch BLEU diversity penalty, so writing a longer reply is a poor way to raise the score. Malformed outputs, answer leakage, and repeated n-grams are dropped. Jaccard and embedding cosine remove near-duplicates. The prompt-only bank then feeds standard OPD from the original student, teacher frozen, advantages restricted to the top-K, under a clipped PPO objective. The question reward does not reproduce that loss.
Math and code each use three student-teacher pairs. Table scores are means of three runs. Recovery is the fraction of the initial average gap that distillation closes.
| Student ← teacher | Student / teacher | Vanilla real bank | A2D | recovery |
| Qwen3-1.7B ← Qwen3-4B-RL-Math | 25.7 / 48.4 | 43.0 (76.4%) | 46.2 | 90.2% |
| R1-Distill-1.5B ← JustRL-DeepSeek-1.5B | 38.8 / 51.7 | 50.5 (90.7%) | 51.6 | 99.9% |
| R1-Distill-7B ← Skywork-OR1-Math-7B | 58.1 / 66.0 | 62.9 (61.0%) | 64.5 | 81.3% |
Against questions sampled from the student with no synthesizer training, the three math averages rise by 5.1, 3.4, and 3.7 points. On Qwen3-1.7B, AIME26 moves from Vanilla's 20.0 to 26.5. On R1-Distill-1.5B, AMC23 moves from 78.1 to 83.6, and the six-benchmark average nearly matches the teacher. KL filtering (the 20% of real questions with the highest disagreement), Evol-Instruct, and R-Zero all finish lower on average. Single benchmarks still break the pattern: on Qwen3-1.7B BRUMO25, Evol-Instruct scores 37.9 and A2D 35.4; on the 1.5B pair, A2D's BRUMO25 is 34.0, below Vanilla's 35.8.
In code, Qwen3-4B distilled from same-size Qwen3-4B-Instruct reaches 39.6 average and 82.9% recovery, against Vanilla at 34.5 and 44.4%. LiveCodeBench hits 35.9, above the teacher's 33.7. Swapping in Qwen3-30B-A3B drops the A2D average to 35.7 and recovery to 32.8%. APPS-Interview still moves from Vanilla's 21.3 to 31.6, and CodeContests from 4.8 to 13.9. Qwen2.5-Coder-3B distilled from the 32B coder averages 21.2, from a 19.7 start and a Vanilla score of 20.2, for 12.3% recovery. A stronger teacher can yield a weaker distilled student.
On τ2-Bench, Qwen3-4B with Qwen3-4B-Thinking, both in non-thinking mode, scores 46.3%, 49.4%, and 30.0% on Airline, Retail, and Telecom. The average is 41.9%, recovery 95.2%. Unoptimized student questions reach 37.3% (79.2% recovery), Vanilla 40.4%, KL filtering 40.9%, and the teacher 43.3%. The student starts at 14.4%, so most of the jump is OPD itself. A2D adds 4.6 points on top.
Across 1,000 questions each, synthetic prompts draw replies of 1,020.7 tokens versus 1,280.2 on DAPO-17k, 20% shorter, with cumulative support-aware signal 181.7 versus 141.7, 28% higher along the whole reply. One paired example goes the other way on the total: a synthetic counting question averages st 0.172 over 1,241 tokens, a real geometry item averages 0.117 over 2,002, and the cumulative scores are 214 versus 234. Both student answers are wrong.
On 1,500 held-out DAPO questions, mean disagreement tracks the drop in probe KL after one OPD step, Spearman ρ = 0.727 (95% CI [0.695, 0.755]). Controlling for length, pass rate, and entropy leaves a partial correlation of 0.386. The top quintile by the full score cuts probe KL by 2.15 times the bottom quintile. Under a shared 24.4M-token budget, five score-ranked banks move six-benchmark accuracy from 41.06% to 43.49% at the fourth bucket; the highest-disagreement bucket lands at 43.11%. Removing compatibility, diversity, or the length penalty drops AIME24/25/26 by 2.1 to 6.6 points from a full reward of 45.0%, 35.8%, and 29.2%. On AIME25 the three drops are 6.2, 6.0, and 6.6.
SFT synthetic data needs a target answer. Self-play question generators are paid by whether the solver is right. OPD reads the teacher's distribution on student prefixes, so the question's job is to choose states. A2D optimizes how much reachable supervision a question exposes. The bank stores no answers, and the downstream update stays fixed.
When a student-teacher pair already exists and good prompts do not, the loop is self-contained: initialize the writer from the student and score with one frozen forward pass. It changes the prompt distribution and leaves token selection and trajectory reweighting alone.
The prompts are not stuck in the domain they were written for. Qwen3-4B trained on code questions moves from 31.38% to 38.60% on four math benchmarks, up 7.23 points, against 5.33 for real-bank Vanilla. R1-Distill-7B trained on math questions moves from 47.00% to 52.40% on three code benchmarks, near the teacher's 52.70%, and passes the teacher on APPS-Intro, 70.1% versus 68.7%.
How large that is depends on the pair. Same-size models with different post-training can recover most of the gap. A much larger teacher leaves headroom that better questions do not close. Pair-specific RL also beats asking the teacher to write the questions. On the code curves, a teacher-initialized writer without RL wins only on the two APPS subsets. Student initialization plus RL reaches about 35% on LiveCodeBench, above the teacher reference of 33.7%, and about 62% on APPS-Intro, still under the teacher's 65.6%.
The reward is one student sample per question, a surrogate. A high score at a position does not guarantee a large learning step, and it is not the downstream gradient. The synthesizer is trained once for the initial pair. Three iterative rounds on code, rebuilding the bank from the updated student each time, move the average of LiveCodeBench and two APPS subsets from 29.03% to 40.53% in round one, then to 40.82% and 40.97%. Two extra rounds add 0.44 points. As the student nears the teacher, questions that combine large disagreement with in-support teacher mass get harder to find. Those two signals are not plotted over training.
The same Qwen3-4B student averages 35.7 when distilled from the 30B-A3B teacher and 39.6 from the 4B teacher. On the 7B math pair, A2D reaches its best checkpoint with fewer downstream tokens: 63.80% versus Vanilla's 61.70% on AIME24, AIME25, and AMC23. On the 1.5B pair the token counts are similar, and the gain is 2.53 points, 54.80% versus 52.27%. Both curves cover only those three math benchmarks. Neither counts the cost of generating and scoring questions.
The cross-domain tests do not separate state coverage from length or other causes. The initialization curves are single runs. τ2-Bench users and judges are GPT-4o-mini, so success rates are tied to that judge. The highest-disagreement bank is less accurate than the fourth bucket. The surrogate and final accuracy do not move together.