Select to Think: SLTop-8 candidates cover 95% of LLM picks, distillation lifts reasoning 24.1%

青稞AI · wechat · 2026-10-15

A TUM PhD student's ICML paper Select to Think (S2T) shows small language models are closer to LLMs than assumed.

Key finding: where SLMs and LLMs disagree, the LLM's chosen token is usually already in the SLM's top-8 candidates — a 1.5B SLM's top-8 covers 95% of a 32B LLM's picks (85% for 0.5B).
Method: instead of having an LLM generate the next step, S2T lets the LLM only select from the SLM's top-K candidates, turning distribution mimicry into a local selection problem. S2T-Local then distills this selection ability back into the SLM via reserved scoring tokens, requiring no external LLM at inference.
Results: a single trajectory approaches 8-path self-consistency and beats greedy decoding by 24.1% on average.

The work suggests new directions for on-policy distillation and efficient reasoning.

More from Research