Allen-Zhu's new paper: Canon layers boost reasoning 2x, even random convolution weights help
teortaxesTex · x · 2026-11-12
Zeyuan Allen-Zhu released Part 4.1 of "Physics of Language Models", introducing Canon layers — lightweight components that compute weighted sums of nearby token representations, promoting horizontal information flow and integrating into Transformers, linear attention, SSMs, or any sequence architecture.
- Using controlled synthetic pretraining tasks (1.3B params, 100B tokens) to isolate core capabilities, the paper presents 12 key results: Canon layers roughly double reasoning depth and improve reasoning breadth and knowledge manipulation
- They lift weak architectures like NoPE to match RoPE, and let linear attention rival SOTA linear models like Mamba2/GDN — validated on synthetic tasks and real academic-scale pretraining
- Implementation is a 1-d causal convolution (kernel size 4) with residual connections via causal-conv1d, adding minimal parameters; the shared screenshot shows even a fixed-random-weights variant already notably enhances performance, analogous to how residuals aid gradient flow
- Songlin Yang noted his observed 19% compute efficiency gain from SConv matches these findings; discussion suggests models like Marin inject many SConv layers for exactly this horizontal information flow
More from Models
- HF researcher: report efficiency-gain metrics, not final benchmark scores — eliebakouch · 2026-11-12
- Terence Tao admits even he was overwhelmed skimming his ChatGPT math session — ChrisGPotts · 2026-11-12
- "Anthropic spent $10B+ on Opus 5.5" is fake news, per community debunk — ns123abc · 2026-11-12
- Domingos: frontier AI capability is rising linearly, +16.5 ECI points/year, no exponential — pmddomingos · 2026-11-12
- Hark launches Pro⁴ plan for power users with 10x the usage of Pro² — adcock_brett · 2026-11-12
- Why Is Inkling 2 Small Trained on So Few Tokens Yet Still Competitive? — georgejrjrjr · 2026-11-12