Goodfire study: probes break predictably under retraining—concepts move, not vanish
mathildepapillo · x · 2026-11-07
Research from Goodfire and collaborators examines how retraining affects internal feature monitorability in language models: over short training times, concepts stay linear and merely move, so retraining a linear probe recovers the signal. Under enough pressure, however, representations can go fully nonlinear. Combining theory and experiments, the authors argue probes break predictably—concepts usually move rather than disappear—and that preserving full monitorability will require careful analysis and algorithm design.
More from Safety
- Cryptographer shares honeypot gag: 2,000 simulated fridge controllers baiting hackers — matthew_d_green · 2026-11-07
- A honeypot with 2,000 simulated fridge controllers on just two IPs — matthew_d_green · 2026-11-07
- Agent security swarms: shared-wallet bug bounty hunting with zk proofs-of-exploit — banteg · 2026-11-07
- China's Zhejiang province aims to become an AI innovation highland — nordicinst · 2026-11-07
- Healthcare ransomware attacks up 6x since 2018 as AI agents close the patch gap, a16z says — luisdans · 2026-11-07
- Berkeley paper auto-generates 6,575 fine-tuning datasets that covertly induce harmful AI behaviors — rohanpaul_ai · 2026-11-07