Stanford paper traces the exact circuit behind in-context learning: an ad-hoc key-value store in the residual stream
CatAstro_Piyush · x · 2026-11-07
A Stanford preprint, A Circuit for In-Context Learning, causally dissects few-shot classification in three 3–4B parameter Transformers using activation patching and head ablations.
Key findings:
- The model builds an ad-hoc key-value store on the fly: middle-layer writer heads write each demo's class into the label position as the key, the label token serves as the value, and later reader heads match the query's class against stored keys before copier heads read out the answer.
- A control loop exists: an early gate checks whether labels are even repeating before allowing retrieval, and dedicated "brake heads" suppress the most recently seen label so the model doesn't lazily parrot it.
- Roughly 60% of class-flip signal passes through reader heads; the circuit transfers across 0.6B–8B models, and Transformers trained from scratch evolve the same gate when retrieval is penalized.
The takeaway: ICL is a gated associative lookup circuit built from a few identifiable, causally verified components—not a mysterious black-box emergence.
More from Research
- Dejan's LLM brand study probes 193k brand mentions for just $391.71 using Gemini — dejanseo · 2026-11-13
- Galaxea Co-founder Hang Zhao's CoRL Keynote: Split Robot Intelligence into System 2/1/0 by Timescale — YuXiang_IRVL · 2026-11-13
- A Summary of the Macy Conferences: The Birthplace of Cybernetics and Early AI Thought — ArtificialOther · 2026-11-13
- vismatch: unified API wrapping 50+ image matching models, 943 stars on GitHub — gabriberton · 2026-11-13
- Stanford's Speculative Self-Distillation cuts training tokens 38-69% with less forgetting — khademinori · 2026-11-13
- EEPO: Adaptive Sample-Then-Forget Rollouts Break Diversity Collapse in RLVR Training — burkov · 2026-11-13