Chain Lens: Training-Free Multi-Token Readout from Model Activations

vgel (co-authored with Raymond Douglas) published "The Chain Lens [interim]" on LessWrong along with a draft paper, proposing chain lens — an interpretability technique that recovers multi-token concepts from model activations without any training — and reporting promising initial results.

Confirmed

• Core method: extends single-token interpretability methods like logit lens and J-lens to multi-token readout: J-lens first transports hidden states to the final-layer basis, then the model's MTP (multi-token prediction) head rolls out token sequences conditioned on that hidden state and the output token — all training-free.
• Three key improvements: contrastive decoding (instead of taking the max-probability token; the biggest gain came from contrastive decoding of generic hidden states), attention masking, and hidden-state reinjection; the authors released the method and pseudocode.
• Composable and easy: J-lens transport and MTP-head sampling combine almost out of the box, and the team got better results with only minor tweaks.
• Results: outperforms J-lens on WorkspaceBench; performs well on workspace-content reading tasks; on a self-built poetry-continuation task, chain lens shows less "confabulation" than oracle lens, though the authors aren't sure why — an open question.

Not yet confirmed

• The reason chain lens confabulates less on the poetry-continuation task is unexplained; the authors explicitly flag it for further research.

Why it matters

• Single-token lenses have long struggled to capture concept structure spanning multiple tokens; chain lens offers a training-free, nearly out-of-the-box path to multi-token readout, and if the results hold, it could become a new tool for auditing concepts inside models and for interpretability research.

2026-11-12 ~ 2026-11-12 · 6 related posts

Full story(3 episodes)→

• Episode 1: Chain Lens: Training-Free Multi-Token Readout from Model Activations(2026-11-12, 6 posts)
• Episode 2: All 1.1M MLP Neurons of Qwen 27B Labeled for Just $5(2026-11-12, 2 posts)
• Episode 3: vgel team explores Loom Lens: sampling MTP heads like a language model(2026-11-12, 2 posts)

• [source] Chain lens: training-free multi-token activation readout beats J-lens 0.42 vs 0.32 — voooooogel · 2026-11-12
• [source] Chain Lens: Training-Free Multi-Token Hidden-State Readout by Reusing MTP Heads — voooooogel · 2026-11-12
• Chain Lens Method Detailed: Contrastive Decoding, Attention Masking, State Re-injection — voooooogel · 2026-11-12
• Contrastive Decoding Against a Generic Hidden State Boosts Chain Lens Most — voooooogel · 2026-11-12
• [source] Chain Lens Confabulates Less Than Oracle Lens on Poetry Task, Authors Puzzled — voooooogel · 2026-11-12

1 near-duplicate retellings: Sauers_