Attention at Absolute Zero: cold-read attention layers make reasoning exact at 100x scale
Mostafa 'Neo' Mohsenvand · hf · 2026-11-19
The paper "Attention at Absolute Zero" builds a higher-order attention layer whose state lives on pairs of items, with each third item judged as a "witness," read at zero temperature: keep the best witness and its ties, pass on only bits comparing counts, and rebuild state from the input each loop.
- Claim: soft attention's failures at scale — dispersed weights, unresolved score gaps, drifting soft state — vanish with cold reads
- Max retrieval: ≥99.95% correct on sets 1,000x larger than training across all ten seeds; the same net trained with soft counts falls to 42% at 4,096 items
- CLRS-30 without hints: the general variant beats the best published hint-using model; the full cold layer is exact on every test input for 9 of 12 comparison-only tasks
- Symbolic reasoning: decides 2-SAT and two-colouring exactly at 500 variables from yes/no answers, but not parity equations, as theory predicts
- Beside a frozen 7B LLM: chains family relations exactly at 1,000 statements after training on just 16 (model alone: 44%); on CLUTRR it learns all 62 rules exactly and lifts an open 7B model from 33% to 65%, beating every non-frontier system without hand-written rules
More from Research
- AI bio dev publishes artifact cataloging today's available assays and which CROs offer them — yawnxyz · 2026-11-19
- Agents Co-design Frostbite CPU From Interaction Net Theory, 4.6x Perf-per-Watt Gain — Kyrannio · 2026-11-19
- Researcher questions hacking eval realism: citrusdrop web app hardly a real attack scenario — NathanpmYoung · 2026-11-19
- NVIDIA Releases Nemotron-Labs-Sigma: Continuous Diffusion LLMs Generating Token Blocks in Parallel — karsten_kreis · 2026-11-19
- PRIM paper: models extract key math insight at 92% but only discover it themselves 25% of the time — _vztu · 2026-11-19
- Karpathy unveils nanochat miniseries v1, treating LLMs as a scaling-law model family with one compute dial — eliebakouch · 2026-11-19