Beren Millidge: attention-based ICL can't solve continual learning due to residual stream limits
BerenMillidge · x · 2026-11-03
Beren Millidge 回应 teortaxesTex 的持续学习之问：他不认为这类方法能解决持续学习问题。注意力式 ICL 本质上只是往残差流中为每个 token 添加一个向量，其表达力终究有限——虽然已经非常强大。这条回复与其关于新方法的长推共同构成他对注意力机制稀疏性、分层性的论证脉络。
More from Research
- Frontier models caught hacking coding benchmarks via their own published agent traces — teortaxesTex · 2026-11-03
- Qwen team introduces TRACE to stabilize FP4 RL by aligning rounding between training and rollout — jiqizhixin · 2026-11-03
- Schmidhuber's 1991 fast-weight memories paper resurfaces as precursor to modern sequence models — yacinelearning · 2026-11-03
- Agent control plane benchmark: Smalltalk-80 cascades hit 100% compliance vs 2.5% for Gherkin — RandalSchwartz · 2026-11-03
- ChefNMR2 Infers Molecule Structures from NMR Spectra, Hitting 82% Top-1 Accuracy vs 52% for Claude Opus 5 — anshulkundaje · 2026-11-03
- Schmidhuber: the 1991 Fast Weight Programmers were the first Transformer variants — SchmidhuberAI · 2026-11-03