Pretraining Study: Hidden States Learn Tasks Before the Readout Expresses Them
Matteo He · hf · 2026-10-17
A new pretraining study, Learning to Read Out, shows that LLM hidden states often encode task information well before the unembedding readout can express it—so behavioral timelines reflect readout learning, not when the model actually acquired the capability.
The method combines three measurements at fixed checkpoints: native readout expression, readout swaps from other checkpoints (margin flips from -0.11 to +1.74), and linear-probe availability. Crosscoder analysis of readout trajectories separates early-decaying, persistent, and late-emerging task features.
Implication: faster readout training closes the learned-but-unexpressed gap earlier, with consequences for interpreting scaling curves.
More from Models
- Reflection AI's open-source Beam-1.0-501B enters Agent Arena top 15 labs, ranked #6 among US labs — arena · 2026-10-17
- Musk: xAI building business-focused Hagglebot, touting up to $6T/year in global savings — elonmusk · 2026-10-17
- Microsoft MAI Base 2 and Meta Muse Spark Open Are the Week's Most Overlooked AI News — The AI Daily Brief · 2026-10-17
- User reports abnormal token consumption: 100% to 0% in a day plus 30k credits — MasterAyolos · 2026-10-17
- Leaker claims K3.1 isn't dropping soon, work predates distillation attacks — ChrisGPT · 2026-10-17
- llama.cpp merges IBM Granite switch support in PR #30676 — jacek2023 · 2026-10-17