Distillation Learns Hesitation, Not Answers: Why Small Models Get Strong and Then Collapse
大模型之路 · wechat · 2026-08-27
A systematic guide to knowledge distillation: the student learns the teacher's full probability distribution ("85% cat, 10% lynx") — that 10% encodes inter-class similarity, Hinton's dark knowledge, amplified via temperature scaling (typically 2-20).
Orthogonal to quantization/pruning: quantization lowers precision and pruning removes redundancy within the same model, while distillation trains a new model; the right order is distill first, then quantize (DistilBERT: 40% smaller, 60% faster, retains 97% of BERT's GLUE score).
Why distilled models collapse on long tasks: offline distillation suffers exposure bias — the student only saw the teacher's polished outputs, so errors compound with O(T²) growth on multi-step generation. On-policy distillation has the student generate first while the teacher gives per-token distributions, reducing error to O(T) with far fewer training rounds than RL. A survey (arXiv:2604.00626) organizes these methods via an f-divergence framework across feedback signal, teacher visibility, and loss granularity.
Two hard constraints: the teacher is the ceiling, and distillation transfers domain-specific response behavior, not the teacher's full knowledge — its errors are inherited and amplified. Per DeepSeek-R1's report, large-scale RL on small models may underperform distillation: RL grows new capabilities in frontier models; distillation cheaply copies existing ones.
Practical path: log real production queries answered by the teacher as distillation data → fine-tune an open-source student → evaluate multi-turn and long responses for degradation → then graduate to on-policy distillation.
More from Models
- Opus 5 uses 5x tokens vs GPT 5.6 for similar task accuracy — abeirami · 2026-08-27
- AdsBench Launches: Kimi K3 Tops AI Marketing Benchmark at $1.42/Task — qinzytech · 2026-08-27
- Zhipu's GLM 3.5 Flash Served 42T Tokens in 6 Days Free on Chinese Chips — bindureddy · 2026-08-27
- Zai's domestic inference cluster hits 100k+ chips; GLM-5.3 runs on custom interconnect — zephyr_z9 · 2026-08-27
- Qwen3.8-Flash-Next generates 112k tokens locally on RTX 3090 — Ok_Ninja7526 · 2026-08-27
- Pruning Qwen3 MoE to 65GB Fits a 180B-Class Model on a 128GB Laptop — EyalToledano · 2026-08-27