Stanford paper traces ICL circuit: few-shot learning is gated label lookup, not learning

rohanpaul_ai · x · 2026-11-05

A new Stanford paper, "A Circuit for In-Context Learning," traces the full mechanism of few-shot ICL across Qwen3-4B, SmolLM3-3B, and Llama-3.2-3B.

Key findings:

The model implements key-value memory over label-token positions: middle-layer attention heads write each demonstration's class into the label token's residual stream, and 90 later reader heads match the query against stored classes and copy the matching label.
A separate set of "brake" heads suppresses the most recently seen label, removing recency bias.
An early-layer signal gates the whole lookup: it opens only when a label repeats — accuracy was 20% with 1 example per class vs 96% with 6. Examples cannot teach a feature the model doesn't already detect.
Causally validated via activation patching and head ablations: 60% of a class flip passes through the readers; roles are conserved across the three models, though which heads fill them varies, and the same structure appears from 0.6B to 8B.

Practical takeaway: in few-shot prompts, repeat each label several times and pick classes the model can already tell apart.

More from Models