Reviving ICA: Fixing Evaluation Bug Makes Classic Method Competitive with SAEs for LLM Interpretability
The furongh team has released new work along with a companion tool, ICA Lens, re-examining Independent Component Analysis (ICA)—a decades-old classical statistical method—for interpreting LLM activations. They found that the evaluation code in an early, influential SAE (sparse autoencoder) paper contained a bug that significantly underestimated ICA's interpretability.
Confirmed
- After correcting the evaluation bug, ICA proves highly competitive with public SAE baselines on interpretability, sparse reconstruction, and concept prediction metrics across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base.
- ICA completes its decomposition per layer in just a few minutes, and interpreting activations requires 7-43x less memory than SAEs.
- The team released ICA Lens, demonstrating how multiple mechanistic-interpretability "lens" tools can work together; in practice, after decomposing Qwen's residuals, features such as cities and France are clearly readable.
- TuhinChakr noted that the ICA baseline bug in that SAE paper had gone unnoticed for three years, and the authors also provided an intuitive video explaining the underlying principles.
Why it matters
- SAEs are the dominant tool in current LLM interpretability research, but come with substantial training costs and memory overhead. If ICA can match SAEs once the evaluation is fixed, the tooling barrier and compute cost of interpretability research could drop significantly.
- This episode also reminds the community that long-cited baseline results may be distorted by code bugs—reproducing and auditing old evaluations is essential to the credibility of the field's conclusions.
2026-10-21 ~ 2026-10-22 · 7 related posts
Primary sources
- ICA Lens finds interpretable LLM directions via non-Gaussianity, no SAE training needed — sarahwiegreffe ·
- UMD team releases icalens: GPU-accelerated ICA fitting with early stopping for interpretability research — sarahwiegreffe ·
- Bug Fix Reveals Classic ICA Rivals SAEs for LLM Interpretability; ICA Lens Open-Sourced — furongh ·
- ICA beats SAEs for interpreting LLM activations with 7-43x less memory, ICA Lens released — feijianghan · 2026-10-21
- [source] Bug Fix Reveals Classic ICA Rivals SAEs for LLM Interpretability; ICA Lens Open-Sourced — furongh · 2026-10-21
- ICA Lens in action: decomposing Qwen residuals reveals clean city/France features — furongh · 2026-10-21
- Bug in SAE paper's ICA baseline fixed after 3 years, making ICA highly competitive — TuhinChakr · 2026-10-22
- [source] ICA Lens finds interpretable LLM directions via non-Gaussianity, no SAE training needed — sarahwiegreffe · 2026-10-22
- [source] UMD team releases icalens: GPU-accelerated ICA fitting with early stopping for interpretability research — sarahwiegreffe · 2026-10-22
1 near-duplicate retellings: furongh