In-context learning is gated lookup of prompt-stored info, not rule fitting, mechanism study finds
burkov · x · 2026-11-12
Andriy Burkov shares an article on ICL mechanics: can models classify sentences with arbitrary labels (foo/bar) from prompt examples alone? Researchers traced the process in three 3–4B base models using controlled prompt-pair interventions on activations and attention heads. The central finding: this form of ICL is a gated lookup of information stored in the prompt — not an observable rule-fitting or weight-update process — filling in how previously identified components combine into a working mechanism.
More from Research
- Anthropic scaled 1,000 Claude agents to formalize Fermat's Last Theorem — SimplyAnnisa · 2026-11-12
- NeurIPS paper gives RNNs provable state-estimation guarantees via control-theory observers — chriswolfvision · 2026-11-12
- NeurIPS paper proves when RNN+MLP can provably recover true robot state — chriswolfvision · 2026-11-12
- Curated 9-paper reading list for LLM quantization beginners, from QLoRA to BitNet — East-Muffin-6472 · 2026-11-12
- A Quick Diagnostic Guide to 4 Common ML Training Errors and Their Causes — cneuralnetwork · 2026-11-12
- DLS runs 1.5x-2.4x faster than tangermeme on A100, verified within numerical precision — anshulkundaje · 2026-11-12