PCA Geometry: Raw Activations vs SAE Feature Strengths
Sauers_ · x · 2026-07-08
Researchers compared the PCA of raw neural network activations (which form a neat circle) with the PCA of Sparse Autoencoder (SAE) feature strengths (which exhibit an irregular shape), revealing significant geometric structural differences. This experiment exposes a potential inconsistency between the SAE feature space and the raw activation space, offering valuable insights for Mechanistic Interpretability research. The visualizations were generated with the assistance of the OpenAI o4.8 model.
More from Research
- HarnessMM cuts agent tokens per task by up to 88% while raising success rates — omarsar0 · 2026-10-26
- Ben Recht: The world needs a truly open language model, and it's easier to build than you think — beenwrekt · 2026-10-26
- New Blog Breaks Black-Box Distillation Into Five Practical Steps — chrisrohlf · 2026-10-26
- DetectorScope turns black-box AI text detectors into auditable ones with sparse autoencoders — _akhaliq · 2026-10-26
- OPST paper shows LLMs self-improve with zero human data, guided only by topic descriptions — _akhaliq · 2026-10-26
- Tencent's HiSTEVE 1.0: hierarchical VLA agent plays Minecraft with native keyboard and mouse, hits 85% SOTA — _akhaliq · 2026-10-26