NYU-Amazon paper: distilling 6 well-picked skills rivals a skill bank 11x larger
rohanpaul_ai · x · 2026-10-10
A new NYU and Amazon paper proposes SGUID: before distilling a skill bank, log which skills yield a steady training signal and drop the rest. Distilling a few skills that keep producing useful signal matches or beats distilling a bank up to 11× larger.
Skills are short written tips (like a counting rule) that a model absorbs by learning from a copy of itself that reads them. Picked by topic match alone, under 25% gave any useful signal across 3 Qwen models.
SGUID keeps only skills that help both early and late in training: with 6 such skills, 3 of 4 models matched or beat the full bank of 30-71 skills on math contest tests; a second round with 3 new skills lifted Qwen3-8B from 64.3% to 66.3%.
More from Research
- QWM method: morphology encoder, adaptive reward normalizer, and latent morphology conditioning — breadli428 · 2026-10-10
- Factuality evals need a rethink: COLM workshop best paper argues current benchmarks are broken — caglarml · 2026-10-10
- Thinking Inertia: LLMs keep reasoning in 99.9% of open-ended answers even with thinking off — rohanpaul_ai · 2026-10-10
- 'The Future of Facts' paper wins Best Paper; co-author calls it year's most underrated — caglarml · 2026-10-10
- Evolution strategies rival policy gradients for LLM fine-tuning: Best Paper Runner-up — caglarml · 2026-10-10
- Datology releases Zephon, a deterministic on-the-fly dataloader born from MosaicML Streaming's legacy — josh_wills · 2026-10-10