Attention circuits lock in within the first 1,000 training steps, Hyperstition study finds

burny_tech · x · 2026-11-12

Hyperstition reports a startling pretraining finding: attention circuits stop meaningfully evolving far earlier than expected.

Two runs differing only in initialization: by the first 1,000 steps, each run's attention resembles its own step-10k pattern more than the two finished runs resemble each other. Initialization matters more than training dynamics — the final pattern is randomly chosen from possible attention basins
Mechanism: existing attention weights act as a strong filter during backprop, creating a positive feedback loop where certain keys gain exponentially more attention while semantically relevant keys stay weakly attended
The team built an oracle to identify weakly attended keys and a natural-gradient method to democratize the backward pass; it broadens attention patterns but worsens validation loss, suggesting winner-concentrating dynamics may also support learning

More from Research