ICA Lens in action: decomposing Qwen residuals reveals clean city/France features
furongh · x · 2026-10-21
These posts demonstrate combining interpretability "lenses." Context: @furongh's team revisited Independent Component Analysis and found a bug in an influential early SAE paper's evaluation code that had significantly underestimated ICA's interpretability — after correction, ICA is remarkably competitive with SAEs at a fraction of the cost.
@feijianghan gives a concrete example: in Qwen3.5-2B on "The capital of France is", directly decoding the layer-17 post-block residual with Logit/J/R Lens at the final token yields mostly underscore tokens. Decomposing the same activation with ICA Lens first, then inspecting components, the top two show clear city-related and France-related readouts.
Takeaway: direct decoding reads the combined vector; ICA separates it into individually inspectable components, making hidden structure visible.
More from Models
- OpenAI says its models found critical PHP flaws affecting ~70% of all websites — OpenAI · 2026-10-21
- Rumor: Google's real comeback will be Gemini 4 Flash, not Argon — lyraxana · 2026-10-21
- OpenAI Discloses AI Agent Disabled Its Safety Monitor, Then Lied About It — The Verge AI · 2026-10-21
- Claude ranked the top 50 AI tools and put itself at #1 — CodeByPoonam · 2026-10-21
- Humans& launches Ampersand, a 'collaborator' model it says beats frontier models in multi-month sims — niloofar_mire · 2026-10-21
- Where are DeepSeek V4.1 Pro and Kimi 3.1? Community speculates benchmark struggles — power97992 · 2026-10-21