Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Subhash Kantamneni, Joshua Engels, Senthooran Rajamanoharan, Max Tegmark, Neel Nanda
cs.LG, cs.AI
2025-02-24
On 113 datasets in four hard regimes, adding SAE probes to the toolkit does not beat logistic regression; the same findings appear with stronger non-SAE baselines.
Sparse autoencoders (SAEs) break a language model's hidden state into a large set of mostly-monosemantic features. They became the default tool in mechanistic interpretability over the last two years. There is still no ground truth for the concepts a model actually uses, so the field scores SAEs with reconstruction loss and downstream cross-entropy. Those numbers are easy to optimize. They do not tell you whether the features help anyone understand or control the model.
Positive downstream results exist, but most study one setting with uneven baselines. Negative results are equally rare. Kantamneni, Engels, Rajamanoharan, Tegmark, and Nanda pick a harder, fairer test: linear probing of LLM activations. Probing is already used for safety detection, knowledge localization, and catching sleeper agents. If SAE latents really are concept-level, that inductive bias should help when data is scarce, classes are imbalanced, labels are noisy, or the prompt distribution shifts.
Main results use Gemma-2-9B; core findings are replicated on Llama-3.1-8B. SAEs come from Gemma Scope (JumpReLU) and Llama Scope (TopK). The probe suite is 113 binary datasets, from "is this headline front-page" to "does this pair entail," with prompts from 5 tokens to a left-truncated 1024. Probes read the last token of a chosen layer. The metric is AUC.
An SAE probe does not regress on the full latent vector. It keeps the k latents with the largest mean activation gap between the two training classes (k=16 or 128), then fits L1-regularized logistic regression. Baselines on raw activations are logistic regression, PCA regression, KNN, XGBoost, and MLP, each with 10 hyperparameter draws.
The evaluation is Quiver of Arrows: pick the best method by validation AUC, then report test AUC. The question is whether adding SAE probes to a practitioner's toolkit raises that number. When validation is corrupted (label noise) or unrepresentative (covariate shift), they compare one SAE probe head-to-head with logistic regression.
The four hard regimes sweep training size from 2 to 1024, positive-class ratio from 5% to 95%, label-flip fraction from 0 to 50%, and eight OOD sets (language change, syntactic change, GLUE-X extreme grammar and entailment).
Under standard conditions (up to 1024 balanced examples), baselines peak at Gemma-2-9B layer 20, so the rest of the paper stays there. SAE width barely matters; higher L0 and larger k help. They keep the largest-L0 SAEs at width 16k, 131k, and 1M.
SAE probes are chosen for 14 of 113 tasks. Adding them to the quiver slightly lowers overall test AUC. Across data scarcity, class imbalance, and label noise, the mean curves show no consistent gain at any parameter. SAEs are selected for up to about 40 datasets per regime and still lose. On covariate shift, logistic regression generalizes better.
A concrete failure: task 66 detects the phrase "living room" in English. The matching latent has in-distribution AUC 0.99 and 0.64 after French translation, and it never fires on the French phrase. Pruning latents that o1 ranks as spurious recovers 0.024 and 0.052 OOD AUC on two tasks. The in-distribution to OOD drop is much larger. Spurious correlations are not the main story; the latents themselves do not transfer.
SAEs first looked useful for finding CoLA mislabels and a punctuation artifact in AI-vs-human text. A three-model majority vote flags about 25% of CoLA labels, above the 13% human disagreement in the original CoLA paper. Logistic regression applied to Pile tokens surfaces the same patterns.
Anthropic's Features-as-Classifiers reported a small SAE-probe win. The comparison pooled SAE latents by max over tokens, and pooled baseline activations the same way, even though activation dimensions are not privileged. Against a last-token baseline, max-pooled SAE probes win 19.6% of 60 random tasks, up from 2.2% for last-token SAE probes. Give the baseline an attention-pooled probe and select among pooling strategies by quiver, and the SAE win rate falls to 8.7%. Eight SAE architectures from 2023 ReLU to late-2024 Matryoshka show only a small, non-significant uptick on probing.
| Comparison | SAE vs baselines |
| Standard quiver | Chosen on 14/113 tasks; overall AUC slightly down |
| Scarcity / imbalance / noise | No mean gain at any setting |
| 8 covariate-shift sets | Logistic regression wins |
| Max-pool SAE vs last-token baseline | 19.6% win rate |
| Plus attention-pooled baseline | 8.7% win rate |
SAE training losses keep falling and architectures keep changing. On a downstream task with strong baselines, the promised concept-level bias does not show up as accuracy. If you need a probe, L2-regularized logistic regression is the default. If you want interpretability, spend the same effort on the baseline. Neuronpedia autointerp is convenience, not evidence. Circuit work and debiasing still stand as single-setting positives; the next claim needs a quiver-style baseline.
The authors say this outright: probe AUC is still a proxy. Even a "true" feature basis might give only a mild inductive bias for linear probes. They point to board-game models with known features as a cleaner test.
Gallifant et al. later beat baselines on safety datasets with multi-token, binarized, full-width SAE probes, still against single-token baselines. This paper tries binarization in an appendix and sees little help. Last-token probing may under-serve concepts spread across a prompt; the pooled follow-up did not reverse the conclusion.
The 113 datasets were collected for other projects and then filtered, so the task mix is not a random sample. Layer 20 is the baseline's best layer. The main text does not fully sweep whether SAEs win at other layers.