Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
Zeyuan Allen-Zhu
NeurIPS 2025 main conference
cs.CL
2025-12-19
Kernel-4 Canon convolutions lift reasoning depth 2-4x in synthetic pretraining, revive NoPE to match RoPE, and lift GLA to rival Mamba2; key trends hold at 1.3B on 100B tokens.
Comparing language model architectures at academic scale is mostly guesswork. At 1.3B parameters on 100B tokens, random seeds and data shuffling alone swing accuracy by 2-4%, and models fail the simplest 2-hop reasoning task: X was born in 1970, Y was born the same year as X, when was Y born? They guess. With every architecture stuck at that floor, differences in harder skills never surface. Industrial-scale training would reveal them, but its cost rules out systematic ablations.
Zeyuan Allen-Zhu (Meta FAIR, Physics of Language Models series) takes the opposite route: decompose intelligence into atomic skills, build one controlled synthetic pretraining task per skill, and compare architectures in that clean room, the way a frictionless plane isolates mechanics.
Five synthetic tasks, one skill each:
Each task runs as a 3x4 mini scaling law (three data scales, four model sizes) with fixed seeds and identical data and step budgets, squeezing out variance.
The discovery made in this playground is the Canon layer: a causal 1-d convolution of kernel size 4 over hidden states plus a residual, h' = h + conv1d(ht, ht-1, ht-2, ht-3). It slots in before attention (A), after the QKV projections (B), before the MLP (C), or inside the MLP (D); any sequence model can take it in a few lines of code. The name comes from the musical canon, where the same melody repeats at fixed delays. On a 1.3B model it adds 0.0063% parameters. Even fixed random weights help; trainable weights help more.
Why does something this small matter? The paper's answer is learning dynamics. Deep skills are learned hierarchically, 1-hop before 2-hop, and lower layers must keep receiving gradient to prepare features only the deeper hops need (backward feature correction). In a language model that correction has to travel diagonally, down layers and back in time. Vertical residuals cover only the first axis; attention covers the second, but slowly and at global cost. Canon opens a local residual channel along time at every layer. Two pieces of evidence: Canon in the first block only recovers a small fraction of the gain, and in some settings the k=1 learning curves are identical with and without Canon while all gains appear at k>=2.
| Setting | Metric | Base | +Canon-ABCD |
| Depo1(K=8), RoPE | accuracy at k=8 | 0-1% | 96-100% |
| Depo2(K=16), RoPE | accuracy at k=16 | 0% | 21-99% |
| Brevo, RoPE | reasoning breadth | baseline | +30% |
| Mano, RoPE | manipulable length | L=10 | matches base at L=13 |
| Capo, RoPE | knowledge capacity | baseline | +10-15% |
| Depo, GLA | reasoning depth | 1-hop | 4-hop |
Roughly 0.5% more trainable parameters buy 2-4x reasoning depth. NoPE (no positional embeddings) starts near zero everywhere; with Canon it beats RoPE+Canon on Depo and clearly beats ALiBi and Hard-Alibi. Keeping RoPE on just 1/4 of dimensions beats both full RoPE and NoPE, good news for length generalization.
The linear-model side is sharper. Removing Mamba2's built-in conv1d drops it to GLA level, while GLA+Canon surpasses Mamba2 and delivers most of what Mamba2/GDN achieve, suggesting many modern refinements largely re-implement conv-style mixing. With every architecture given full Canon, linear models hold about 40% more knowledge but Transformers reach 2-4x their reasoning depth; the bottleneck is cumulative compression and retrieval error, not state size.
Real 1.3B/100B-token pretraining reproduces the main trends: Canon lifts NoPE and GLA to match RoPE and Mamba2/GDN respectively, and every model still fails 2-hop reasoning even in 100-token contexts.
Half methodology, half architecture finding. The synthetic playground makes rigorous architecture science affordable and reproducible; Canon itself is a plug-in component for any sequence model, and along the way it recovers knowledge capacity lost to gated MLP and MoE designs. For anyone designing architectures, the claim that most of Mamba2/GDN's advantage can be reproduced by GLA plus a small convolution deserves serious attention. The paper is also explicit that Canon buys learning speed, not representational power: given unlimited training, vanilla RoPE would eventually get there.
Author-stated: with a fixed training budget, results measure speed of skill acquisition, not absolute capability; the deepest structural task (cfg3k) stays out of reach; naive Canon-ABCD costs 12.4%/14.1%/20.8% forward/backward/generation runtime (Canon-AC drops to 5.8%/5.8%/7.0%). Real academic-scale pretraining stays noisy enough that many comparisons are statistically unreliable, leaving only a few stable trends. Larger-scale results (1-8B on 1-2T tokens) are summarized as consistent with the paper but not presented in detail. The contribution statement says every idea, line of code, experiment, and the entire manuscript came from a single author, so independent replication is so far left to the community.