One prompt is enough: distillation from a single query hits 71.5% of full-data gains
burkov · x · 2026-09-06
Researchers from Tsinghua, UCAS, Northeastern, UIUC and Johns Hopkins show on-policy distillation — where a teacher corrects the student at every token — recovers most of the improvement from 17,000 training queries using just one query, reaching 71.5% of the state regions visited by full-data training. A single prompt spawns many prompt-response "states," each giving the teacher another correction opportunity; 16 semantically diverse queries extend coverage further.
More from Research
- JEPA may keep unpredictable features: new paper gives set-level characterization — hisspikeness · 2026-10-23
- GAFU Uses Generation-Side Representations to Boost Fine-Grained Visual Understanding in Unified Multimodal Models — Tengju Ru · 2026-10-23
- NTU Study: Learned Unmasking Policies Improve Speed-Accuracy Trade-offs in Diffusion Language Models — nanyang-technological-university-singapore · 2026-10-23
- NBER releases 'The Economics of Transformative AI' edited by Agrawal and Brynjolfsson — JMateosGarcia · 2026-10-23
- Swapping only the eval harness moves results 30%: designing agent evals like a scientist — AI Engineer · 2026-10-23
- Jane Street and NVIDIA speed up Mamba-2 with custom cuTile kernels — blelbach · 2026-10-23