Peking University finds models "know" answers but can't say them: the Readout Bottleneck
jiqizhixin · x · 2026-09-26
A study from Peking University and YIXIN AI Lab (accepted to EMNLP 2026 main conference, 15.4% acceptance rate) asks: when a model fails a logic problem, does it truly fail to reason, or does it already know the answer but fail to express it?
Opening the black box, the researchers find the correct answer is already encoded in the model's hidden states, but a systematic bias in the output layer "covers it up" when those states are converted into candidate scores. They name this phenomenon the Readout Bottleneck.
Mechanistically, even when the model's first-label-token score still holds a slight advantage, accumulating log-probabilities across the full candidate string amplifies the error — accuracy collapses to 0.333, i.e. pure three-way guessing. The effect persists on an instruction-tuning-free base model: probing recovers 87.3% on Qwen3.5-9B-Base, while sequence scoring drops sharply.
More from Research
- Scandinavian Journal of Statistics editors reflect on publishing in the age of generative AI — michaelchchoi · 2026-11-11
- Proving LLM backdoor-safety without revealing weights: ZKML over formal methods — DimitrisPapail · 2026-11-11
- Claude Opus 5.5 runs an architecture search and finds AttnRes-Canon, a new top Transformer design — SonglinYang4 · 2026-11-11
- 5 research habits worth knowing: read papers backwards, reproduce first, wire coding agents via MCP — cneuralnetwork · 2026-11-11
- Running Qwen2.5-Coder-3B on Android: Jupyter + Pydantic Evals on-device eval pipeline — alex_b3b · 2026-11-11
- cloneofsimo releases a video explaining determinantal point processes — cloneofsimo · 2026-11-11