NYU and Amazon's SGUID: 6 selected skills match distilling a skill bank 11x larger

rohanpaul_ai · x · 2026-10-10

An NYU/Amazon paper (arXiv 2610.12367) tackles which skills are worth distilling into LLMs.

Skills are reusable procedural guidance added at inference (e.g., a case-counting rule) that can substantially boost downstream performance
Key finding: in on-policy distillation, fewer than 25% of retrieved skills provide useful distillation signals
SGUID keeps only skills that consistently yield effective learning signals during training, then distills those
Results: across four models from the Olmo and Qwen families, distilling just 6 selected skills matches or exceeds full-bank distillation (banks up to 11x larger) on three of four models — and on all four after a second round distilling 3 newly selected skills
The method supports stable model-skill co-evolution: candidate banks are curated from the updated model's rollouts each round

More from coding & agent