Layer-wise heatmaps show trained models 'compile' patterns into planning steps
Sauers_ · x · 2026-10-21
An interpretability observation using layer-wise activation heatmaps on a concept never seen in training ("Gnomes"):
- The trained model develops internal structure matching the requested pattern, stable even when the exact plan is prefilled—suggesting planning lets the model 'compile' arbitrary patterns into layer-by-layer steps.
- Notably, the original model with thinking off already does decently, with messier but partially matching internal activations.
- The figure compares requested pattern, trained model, trained model with prefilled plan, and original model without thinking.
More from Research
- Single neurons can steer LLM behavior — echoing a 2007 finding in real brains — HistedLab · 2026-10-21
- Dev rebuilds real Swiss terrain at 0.5m detail with 3,800 lines of code, no game engine — techartist_ · 2026-10-21
- UnsolvedMath dataset hits 17,614 open problems for LLMs, drawing mathematicians' ire — tak3sh8 · 2026-10-21
- mini-Lean-agent: bash-only coding agent solves 12/12 Putnam 2025 at $1.28 per problem — wellecks · 2026-10-21
- Rethinking vanilla pixel diffusion transformers: capacity limits and a high-frequency deficit in JiT — CSProfKGD · 2026-10-21
- Visa trouble may cost authors their accepted EMNLP paper over a no-show — OtherwiseFig4505 · 2026-10-21