First-hand: two SAE features both labeled 'user turn start' encode user-type differences
voooooogel · x · 2026-10-18
Following up his SAE critique, voooooogel shares a first-hand example: while RL-training a model, he watched SAE feature distributions shift at the start of a user turn from one feature to another — yet both features were labeled 'START OF USER TURN' with identical top activating tokens.
Inspecting max activating examples revealed a real difference: one user type roleplayed or requested code in a brusque way, the other was more 'normal' — subtle and hard to characterize. He doubts a Claude agent could find this without burning heavy tokens on experiments, and infers that detectable-from-examples structure like this signals real information — supporting his point about auto-interp struggling with linguistically inexpressible distinctions.
Related event: Researcher Questions SAE Interpretability with RL Training Evidence(2 posts)→
More from Research
- Detecting Speech AI Hallucinations from Attention Patterns: Tested on Qwen-2-Audio and Voxtral-3B — burkov · 2026-10-18
- Teen solves years-old math problem in a week with AI and Lean, writes 15-page paper — IgorCarron · 2026-10-18
- SFT + simple search tool lets Qwen3-4B match a ~1T-param model on diagnosis benchmark — Tim_Dettmers · 2026-10-18
- Automating Experiments Makes Premise Choice the Real Bottleneck in Recursive Self-Improvement — Gauri_the_great · 2026-10-18
- Papers with Code Launches Following Feed to Track AI Tasks, Benchmarks and Authors — NielsRogge · 2026-10-18
- Moritz Hardt: AI benchmarks are fundamentally broken but fixable — asusarla · 2026-10-18