Probe Detection Boils Down to a Single Attention Head Reading Tiny Hidden-State Differences
Sauers_ · x · 2026-10-18
Sauers traces the generalization failure to the model using tiny differences in the user token's hidden state to detect an active probe: swapping the probed model's activations at the user token alone into an unprobed model suffices to flip behavior, entirely via one attention head in the final layers. Visuals show training shrank head 6's key at user from length 70 to 6.6, enlarging the probe's change (0.5→4.2, 0.4°→37°), shifting attention from </think> (13%) to user (100%) and flipping the answer to Yes.
More from Research
- Blocking a single protein lets mouse retinas regenerate naturally — Promptmethus · 2026-10-18
- Neuroscientist Giorgio Gilestro: Brain Analogies Mislead on LLM Architecture — solyarisoftware · 2026-10-18
- APEX-MEM: property graph + append-only events for LLM long-term memory — burkov · 2026-10-18
- SAE researcher questions sparse coding premise and Claude-based neuron autolabeling — voooooogel · 2026-10-18
- Redditor claims homegrown hybrid architecture hits 15.85% on ARC-AGI-3, open-scores repo for audit — LooseSwing88 · 2026-10-18
- IdeaLens detector claims to tell whether a paper's core idea came from AI or a human — skdh · 2026-10-18