LessWrong rebuttal argues SAE 'sanity check' paper's toy setup is hostile, not easy
burny_tech · x · 2026-11-01
Celene Nightingale published a LessWrong post disputing the methodology of Korznikov et al.'s paper claiming SAEs recover only 9% of ground-truth features in a synthetic toy setting. She argues the setup is not the "simplest, fully aligned" environment the original authors imply, but is actively hostile to SAEs in ways real settings are not, so the 9% figure can't be read as evidence SAEs fail to recover true features. The post marks the latest round in the field's growing pessimism about SAE work, focusing on the paper's first question (can SAEs recover real features?) rather than the random-baseline comparison.
More from Research
- Control Vectors experiment: letting an agent manage its own activations — sloppenheimer · 2026-11-01
- Claude designed scorpion toxin binders: 17 of 211 lab-tested designs bound — shoumikgoswami · 2026-11-01
- Hugging Face open-sources Carbon DNA foundation models with full training recipes — LoubnaBenAllal1 · 2026-11-01
- Carbon-X: midtrained Qwen3-14B beats every open LLM on DNA benchmarks — LoubnaBenAllal1 · 2026-11-01
- RL fails without midtraining: Qwen3-14B stays at chance on FrontierVEP — LoubnaBenAllal1 · 2026-11-01
- FineDNA midtraining lifts DNA scores without degrading English — LoubnaBenAllal1 · 2026-11-01