Goodfire's Neural Geometry series: steering models along manifolds
burny_tech · x · 2026-11-03
Interpretability startup Goodfire published the 'Neural Geometry Series,' arguing that neural networks develop rich geometric structure in their activations — days of the week form circles, colors form an HSL manifold — and that mapping this geometry is key to understanding and controlling AI.
Five essays (May 2026):
- Opening post: the case for neural geometry as a frontier of AI understanding and control.
- Steering along manifolds: curved-manifold steering produces cleaner, more targeted behavior changes than linear steering vectors.
- A geometric calculator: a general-purpose addition module in Llama 3.1 8B operating over circular number representations.
- Can SAEs capture neural geometry?: an unsupervised pipeline for discovering manifolds from SAE features.
- Stories over time: how LLM representations of narrative evolve across a prompt or conversation.
More from Research
- Victor Taelin floats labs replacing backprop with models editing their own weights at scale — arthurcolle · 2026-11-03
- LLM paper claims last ~15 months vs ~75 years for general science, researcher estimates — patpat_mit · 2026-11-03
- LocalFold update brings BioEmu2, Mol-1, AF3x crosslinks to browser-based protein folding — CatAstro_Piyush · 2026-11-03
- kalomaze conjectures deep sequence models could mimic restricted training families like rank-1 attention LoRAs — kalomaze · 2026-11-03
- Zyphra's ZHA brings subquadratic attention via hierarchical tree-structured indexer — yacineMTB · 2026-11-03
- kalomaze: In-context learning is an incidental byproduct of next-token prediction, not deliberate design — kalomaze · 2026-11-03