Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning
Yinghui He, Ling Yang, Jiarui Liu, Yongjin Yang, Lechen Zhang, Yingcheng Wu, Zhenfei Yin, Mengdi Wang, Sanjeev Arora
cs.CL, cs.LG
2026-08-06
Switching skills mid-chain (math then scheduling) is a weakness apart from per-skill ability; using Skill Entropy as a GRPO reward lifts Qwen3-4B from 34.4% to 68.4% on the new Skill²-Bench.
A long-horizon reasoning problem often needs several skills in one chain: do a math derivation, use the result to plan a schedule, then use the schedule for information extraction. The authors call these cross-skill long-horizon tasks, multi-step tasks where each step needs a different skill and depends on the previous output.
Existing benchmarks mostly test one skill at a time and lack a ruler for how well a model switches between skills. The striking part is that frontier models do well on single-skill benchmarks yet falter on these composite tasks, even when each component skill is one they handle alone. Skill switching looks like a capability orthogonal to domain competence, and it does not emerge from stronger per-skill training. The common failure: at later steps the model reuses the previous step's skill and answer modality instead of switching to the one the current step needs.
The authors first build a ruler, Skill Entropy SkE(sa, sb), measuring how hard it is to switch from skill sa to sb. It is directional; reversing the order can change the value. Operationally it is a smoothed ratio: the average of each skill's solo accuracy, divided by the accuracy of doing them as a two-step chain. A value near 1 means chaining adds little difficulty; larger means harder to switch.
Computing all pairs directly is infeasible (about 310,000 ordered pairs). The authors factor it: SkE(sa, sb) is approximated by the average cost of leaving sa times the average cost of landing on sb. A fixed reference model (Claude-opus-4.7) scores these, keeping the ruler consistent across all evaluated models.
On this ruler they build Skill²-Bench: 558 skills across 9 domains (math, science, coding, logic, information extraction, and planning verifiable; creative writing, context retrieval, and instruction following open-ended). Each task's entropy is the average along its skill sequence, split into low, medium, and high levels. Verifiable domains use deterministic scorers (symbolic equivalence, sandboxed tests, multiple-choice match); open-ended domains use a rubric with an LLM judge (also Claude-opus-4.7), correlating with humans at Pearson r = 0.84-0.91.
The evaluation covers 8 frontier and 4 open-source models. The core finding is a skill-switching gap: the same skill done inside a cross-skill task drops 4 to 13 points versus solo (4% to 10% for frontier models), declining nearly monotonically with task entropy. The largest drops are on planning. On wrong-domain steps, per-step accuracy (32%-57%) is roughly half that on right-domain steps (63%-79%). One line: picking the wrong skill at a step roughly halves its accuracy.
| Model | Solo to cross-skill drop |
| Qwen3-4B | -12.6 |
| GPT-5.4-mini | -10.0 |
| Claude-opus-4.7 | -4.3 |
The authors then turn the ruler into a training signal, Skill-Entropy RL: at each step the model must report which skill it used (forced through a structured format) before answering. The reward is 0.7 times answer score plus 0.3 times a skill-entropy score measuring how well the predicted skill sequence aligns with the gold one. On Qwen3-4B-Instruct, Skill²-Bench rises from 34.4% to 68.4%; on Qwen3-1.7B, from 14.6% to 40.1%, beating GRPO, STAT, and other baselines. The same pipeline applied to off-the-shelf OpenR1-Math also wins, best on all six math benchmarks.
This gives the field a reusable number. Skill Entropy turns the vague complaint that a model is bad at skill switching into something measurable and tunable, and the same number works as a GRPO reward that roughly doubles a small model's cross-skill score. For anyone training long-horizon reasoning, it is a cheap, reusable training signal.
The paper has no Limitations section (the word does not appear). Scattered bounds: skill entropies are computed under Claude-opus-4.7 as reference; recomputing under Gemini-3.1-pro or GPT-5.5 gives partition overlap above 80% with consistent trends, but not identical. The skill bank is LLM-labeled with manual review, granularity kept comparable by humans. Pairwise entropy is a multiplicative approximation, not exact. The RL reward requires a gold skill sequence, so off-the-shelf data needs extra annotation first. RL is run only on small models (at most 7B), not at frontier scale. Open-ended scoring depends on an LLM judge, with no judge-swap robustness check.