HAPAX training strips induction copying yet abstractive ICL still emerges
davidbau · x · 2026-10-26
The HAPAX paper (EMNLP 2026 Main) from David Bau's group at Northeastern tests whether induction heads underlie all in-context learning.
- HAPAX is a training regime that excludes loss on tokens predictable by induction heads, reducing the training pressure to learn rote copying.
- Despite substantially less inductive copying, fewer and weaker induction heads, and no characteristic phase changes, abstractive ICL still emerges, reaching accuracy comparable to vanilla training.
- Conclusion: the developmental link between induction circuits and abstractive ICL is weaker than previously hypothesized — rote copying is not a prerequisite for harder skills, and the copying-free model even translates slightly better.
Paper, code, and model weights are public.
More from Research
- CoRL paper: one-step inference matches multi-step for large VLAs, 1.6-4.8x faster action generation — kaixhin · 2026-10-26
- Why biology is hard to simulate: universally sloppy parameter sensitivities, per classic PLOS paper — rbhar90 · 2026-10-26
- LessWrong post argues FDT is difficult-to-impossible to specify due to logical counterfactuals — jessi_cata · 2026-10-26
- Bitter Lesson misread: scaling won't brute-force biology, researcher argues — anshulkundaje · 2026-10-26
- Chain Theory: rebuilding physics from quantum information, AI finds 137 fine-structure-constant expressions — skdh · 2026-10-26
- HKU/ZJU/Fudan with Lightspeed launch HiSTEVE: hierarchical VLA agent playing Minecraft via native mouse and keyboard — tencent · 2026-10-26