SAE researcher questions sparse coding premise and Claude-based neuron autolabeling
voooooogel · x · 2026-10-18
Interpretability researcher voooooogel raises two critiques of SAE-based interpretability:
- Questionable sparsity premise: SAEs working poorly may itself be a signal — imposing a sparse structure may be unwarranted if neurons already hold structured semantic information and aren't sparse.
- Poor autointerp quality: labeling SAE features with Claude agents yields gross errors on nearly every inspection, plus subtle distinctions that are hard to express linguistically — making him skeptical of the 'just throw Claude at it' labeling program unless enormous compute is being spent on interp agents.
He remains cautiously optimistic about an upcoming technical post but questions SAE's theoretical foundation.
Related event: Researcher Questions SAE Interpretability with RL Training Evidence(2 posts)→
More from Research
- Taste-Bench shows AI agents make poor long-term strategic choices, top model under 60% — The Decoder · 2026-10-18
- ~1,700-model study: diffusion LMs need only 3-8x AR compute, not the folkloric 16x — LucaAmb · 2026-10-18
- GPT-2 KV cache from scratch: 1000x fewer FLOPs but only 6x latency gain, measured — atronos_kronios · 2026-10-18
- Blocking a single protein lets mouse retinas regenerate naturally — Promptmethus · 2026-10-18
- Neuroscientist Giorgio Gilestro: Brain Analogies Mislead on LLM Architecture — solyarisoftware · 2026-10-18
- APEX-MEM: property graph + append-only events for LLM long-term memory — burkov · 2026-10-18