Distilling Qwen 27B into 4B: three distillation methods tested to break a 58% wall
Prompt Engineering · youtube · 2026-10-29
Prompt Engineering follows up its RL experiment (Qwen 4B stuck at 58.4%) by using Qwen 27B as a teacher across two DGX Sparks, testing three distillation approaches: training on teacher answers, on teacher probabilities (which added almost nothing), and on-policy distillation where the 27B grades every token the 4B writes (per Thinking Machines' method).
The video covers why RL plateaued, a six-method comparison chart, a 4-tries test, and two practical gotchas: tokenizer mismatches and vLLM memory issues. Code and the SmolDataEnvs dataset are open-sourced.
More from Models
- Pedro Domingos: Claude is 'pretty useless' for code never written before — pmddomingos · 2026-10-30
- AI Code Security Finally Improves: Veracode Benchmark Pass Rate Jumps From 56% to 62% — WeldPond · 2026-10-30
- Anthropic exec confirms $500 plan can nearly saturate most bug hunters' agent usage — rez0__ · 2026-10-30
- 40 days of coding-agent telemetry show a provider silently swapped model weights — Jipok_ · 2026-10-30
- Local Gemma 'completely good enough' for non-coding tasks, designer finds — michalmalewicz · 2026-10-30
- Gemini Pro silently downgraded to Flash all day citing high demand — General_Internet_511 · 2026-10-30