Goodfire's mechanistic shaping turns model internals into training feedback to fix shortcuts
mathildepapillo · x · 2026-11-06
Interpretability startup Goodfire introduces mechanistic shaping, a mathematically rigorous framework that converts model internals into training feedback, controlling which concepts a model relies on during training.
Demonstrated applications (figures show chest X-ray feature labeling):
- Mitigating reward hacking
- Removing shortcut features in chest X-ray diagnosis — separating genuine evidence (chest drains) from spurious cues like ECG electrodes and image borders via feature scoring for pseudo-labeling
- Reducing sycophancy and harmfulness
The authors acknowledge a known risk: training against a model's internals can make the model harder to monitor. The framework claims to work across domains, problems, and modalities.
More from Safety
- Miles Brundage cites METR: Anthropic will pack 4 months of progress into 2 before Congress — nitarshan · 2026-11-06
- Reporter's ChatGPT 'sharpening' of news photo reignites debate over AI edits altering meaning — haydenfield · 2026-11-06
- Ajeya Cotra: redacted evidence undermines AI risk verification and safety cases — ajeya_cotra · 2026-11-06
- Vinod Khosla: AI plus quantum computers will break most encryption before 2030 — rohanpaul_ai · 2026-11-06
- Yale study: ChatGPT recommended a Senate candidate in 79% of political chats — luisdans · 2026-11-06
- Compound Security: open-source skill for agent-driven vulnerability patching — hamostaf04 · 2026-11-06