How rollout staleness hurts on-policy distillation, and how to build a faster pipeline
burkov · x · 2026-11-17
Andriy Burkov highlights a systematic study of staleness in on-policy distillation pipelines. Post-training often spends more time generating long reasoning responses than updating weights; running generation, teacher scoring, and training in parallel improves hardware utilization but means the student trains on data from an older version of itself.
Key points:
- Storing or transferring teacher scores for every possible next token is prohibitively costly, so systems cache scores for only a limited set of actions, constraining distillation objective design
- The work systematically tests how different distillation objectives and teacher-feedback caching strategies degrade as rollout data gets stale
- Experiments use Qwen3 student models with a Qwen3 teacher on a filtered dataset (57.6k samples per the truncated post)
- The analysis points toward faster pipelines that don't sacrifice model quality
More from Infra
- Engineer runs Gemma 3 270M entirely inside an office laser printer at 60s per token — osanseviero · 2026-11-17
- Older GPUs abandoned by new inference engines: are they still worth keeping? — pipecanon · 2026-11-17
- Blogger argues total compute controls create blind spots and accelerate adversaries — TinfoilTricorn · 2026-11-17
- SK hynix's Nvidia revenue doubled to $24B, but US customer concentration is the real risk — tengyanAI · 2026-11-17
- macOS TCP Stack Breaks After Exactly 49.7 Days of Uptime Due to 32-bit Timestamp Overflow — DanielLockyer · 2026-11-17
- Running a 2M-param embedding model on an ESP32 with no_std Rust at 830 embeddings/s — qdrant_engine · 2026-11-17