NeurIPS paper finds LLM representations get sparser as tasks get harder, and uses it to boost ICL
zhaoran_wang · x · 2026-10-27
A NeurIPS 2026 paper, Farther the Shift, Sparser the Representation (Mingyu Jin, Zhaoran Wang et al.), systematically probes what happens inside LLMs as inputs get harder.
Key findings
- Across four difficulty dimensions — harder reasoning problems (MATH-500, DeepMATH-103K), more answer choices, knowledge conflicts, longer contexts — the last hidden state becomes systematically sparser (|r|>0.89), consistently across models and tasks.
- Training toy Transformers from scratch shows the pattern is learned, not architectural: early training drives concentration-based sparsity that later gives way to dense representations for mastered inputs, while OOD inputs stay sparse.
Application: The authors propose Sparsity-Guided Curriculum In-Context Learning (SG-ICL), which uses representation sparsity to select few-shot demonstrations matching query difficulty, outperforming standard CoT and Auto-CoT on MATH-500. Code is open-sourced at github/MingyuJ66/SparsityLLM.
More from Research
- Model breeding begins: neuron combo makes Qwen believe it's Cookie Monster — yacineMTB · 2026-10-27
- Sakana AI's RSI Lab debuts in Japan: world models, Sawara and AI Scientist demos — Stefania_druga · 2026-10-27
- EMNLP paper uses sparse autoencoders for text-based causal confounding adjustment — anjalie_f · 2026-10-27
- Volta releases Blackwell kernel verifier letting LLMs generate PTX directly, skipping compiler layers — Azaliamirh · 2026-10-27
- Johns Hopkins NLP Group Meets at EMNLP 2026 in Budapest — mdredze · 2026-10-27
- Poking a single neuron in Qwen turns it into an alien guiding civilization — MarcJSchmidt · 2026-10-27