kalomaze conjectures deep sequence models could mimic restricted training families like rank-1 attention LoRAs
kalomaze · x · 2026-11-03
kalomaze 延续其对 ICL 与训练步关系的讨论，提出猜想：一个足够深的序列模型或许能在对数概率偏移意义上模拟某些受限训练家族（如仅注意力的 rank-1 LoRA）在序列化样本上的行为。他认为要做到这一点，需要刻意的 KL 蒸馏才能引出「ICL 表现得像对独立样本执行训练步」的现象，而目前社区并未有意朝这个方向推进。
More from Research
- Starfall: Elon's orbital drug lab plans Falcon 9 launches and Pacific splashdowns for better crystals — ZeroStateReflex · 2026-11-03
- Bench Summit Consensus: Good Tooling for Grokking Traces Is Scarce — MeganRisdal · 2026-11-03
- LLM coding agents write from-scratch LP solvers averaging ~10x faster than general-purpose ones — sh_reya · 2026-11-03
- Schmidhuber reups 'Miraculous Year 1990-91' essay claiming GANs, Transformers, pretraining roots — SchmidhuberAI · 2026-11-03
- Only recent models seem to know "weight surgery" — are labs building Bayesian weight priors? — Tkaraletsos · 2026-11-03
- Victor Taelin floats labs replacing backprop with models editing their own weights at scale — arthurcolle · 2026-11-03