研究:注意力电路早在训练初期就定型,最终模式近乎随机选定
burny_tech · x · 2026-11-12
Hyperstition reports a startling pretraining finding: attention circuits stop meaningfully evolving far earlier than expected.
- Two runs differing only in initialization: by the first 1,000 steps, each run's attention resembles its own step-10k pattern more than the two finished runs resemble each other. Initialization matters more than training dynamics — the final pattern is randomly chosen from possible attention basins
- Mechanism: existing attention weights act as a strong filter during backprop, creating a positive feedback loop where certain keys gain exponentially more attention while semantically relevant keys stay weakly attended
- The team built an oracle to identify weakly attended keys and a natural-gradient method to democratize the backward pass; it broadens attention patterns but worsens validation loss, suggesting winner-concentrating dynamics may also support learning
「研究」频道最新
- HipKittens 一周年:团队持续攻坚 AMD GPU 软件生态破 CUDA 护城河 — simran_s_arora · 2026-11-12
- 世界模型团队争抢带操作标签的游戏数据：动作才是稀缺的那一半 — KnowledgeOk7634 · 2026-11-12
- Uno 用离散扩散让 LLM 推理提速至 3 倍,作者将开线上讲座 — burny_tech · 2026-11-12
- Pro-0 触觉手套零机器人数据训练灵巧手策略，亮相 CoRL — chris_j_paxton · 2026-11-12
- Physical Intelligence 拆解 Fable vs π0.5：零样本 LLM 已胜微调 VLA — paigeinsf · 2026-11-12
- 零样本 LLM Fable 跑赢微调 VLA：记忆任务 64 分 vs 7 分 — paigeinsf · 2026-11-12