kalomaze conjectures ICL may mimic training steps via deliberate KL distillation
kalomaze · x · 2026-11-03
kalomaze proposes a technical conjecture: a sufficiently deep sequence model might be able to behave like a restricted training family (e.g., rank-1 attention-only LoRAs) in terms of logprob shifts given sequenced examples. He suggests deliberate KL distillation would be needed to elicit the phenomenon that "ICL literally behaves like training steps on independent examples." Still a conjecture with no empirical backing.
More from Research
- Qwen team introduces TRACE to stabilize FP4 RL by aligning rounding between training and rollout — jiqizhixin · 2026-11-03
- Schmidhuber's 1991 fast-weight memories paper resurfaces as precursor to modern sequence models — yacinelearning · 2026-11-03
- Agent control plane benchmark: Smalltalk-80 cascades hit 100% compliance vs 2.5% for Gherkin — RandalSchwartz · 2026-11-03
- ChefNMR2 Infers Molecule Structures from NMR Spectra, Hitting 82% Top-1 Accuracy vs 52% for Claude Opus 5 — anshulkundaje · 2026-11-03
- Schmidhuber: the 1991 Fast Weight Programmers were the first Transformer variants — SchmidhuberAI · 2026-11-03
- Liquid AI's Antidoom cuts reasoning model doom loops from 10.2% to 1.4% with FTPO — maximelabonne · 2026-11-03