kalomaze: In-context learning is an incidental byproduct of next-token prediction, not deliberate design
kalomaze · x · 2026-11-03
kalomaze 在与 Beren Millidge 的讨论中提出：当前的上下文学习（ICL）并非刻意设计，而是下一词预测（NTP）在骨干网络获得更多条件信息时的被动副产物。他称尚未见过有人尝试把分类器在多个独立 SGD/Adam 步（即连续序列训练的不同时点）上的 logits 做蒸馏。在后续补充中他进一步猜想：一个足够深的序列模型或许能模拟受限训练家族的行为，但需要刻意的 KL 蒸馏才能引出「ICL 表现得像对独立样本的训练步」这一现象。
More from AGI Musings
- Schmidhuber reups 'Miraculous Year 1990-91' essay claiming GANs, Transformers, pretraining roots — SchmidhuberAI · 2026-11-03
- LLM paper claims last ~15 months vs ~75 years for general science, researcher estimates — patpat_mit · 2026-11-03
- Fields Medalist Richard Borcherds on AI and math: 'AI is already better than me' — TOEwithCurt · 2026-11-03
- After visiting Chinese AI labs: the gap with US frontier labs is widening — pstAsiatech · 2026-11-03
- The Blindsight 'Conspiracy': Are Blindsight Patients Just Pretending? — theshawwn · 2026-11-03
- Paul Graham: the scariest part is calling this a normal year — paulg · 2026-11-03