Attention head acts as a switch: mechanistic interp case flips model answer from No to Yes

CatAstro_Piyush · x · 2026-10-18

• Researcher Sauers shares an interpretability case: an attention head behaves like a switch — without a probe it attends to and pushes the answer toward "No"; with the probe it attends to the user token and pushes toward "Yes".
• Layers 34 and 35 use cancellation, like zeroing a scale: the model adds a component opposite to the normally large key at the user token, canceling it out so only the probe's tiny change remains, which is then normalized into a large change that shifts which token is attended to — making the model say yes.
• The reposter calls this a "hard mode model organism," challenging mechanistic interpretability tools to detect it.

Original post →

More from Research

• Detecting Speech AI Hallucinations from Attention Patterns: Tested on Qwen-2-Audio and Voxtral-3B — burkov · 2026-10-18
• Teen solves years-old math problem in a week with AI and Lean, writes 15-page paper — IgorCarron · 2026-10-18
• SFT + simple search tool lets Qwen3-4B match a ~1T-param model on diagnosis benchmark — Tim_Dettmers · 2026-10-18
• Automating Experiments Makes Premise Choice the Real Bottleneck in Recursive Self-Improvement — Gauri_the_great · 2026-10-18
• Papers with Code Launches Following Feed to Track AI Tasks, Benchmarks and Authors — NielsRogge · 2026-10-18
• Moritz Hardt: AI benchmarks are fundamentally broken but fixable — asusarla · 2026-10-18

Research channel →