Goodfire's Mechanistic Shaping Lifts Worst-Group Accuracy From 51.6% to 97.7%
burny_tech · x · 2026-11-10
Goodfire released a paper, "Shaping & Monitoring Models through their Internal Representations," proposing Mechanistic Shaping: a training-time framework that constrains which concepts a model expresses and which concepts it relies on when deciding (e.g., no harmful output, or no reliance on a candidate's gender in résumé screening).
Key results
- Tested across vision, medical imaging and language, up to 32B parameters, and classification, SFT, DPO and GRPO training paradigms.
- On a biased image-classification benchmark, worst-group accuracy rises from 51.6% to 97.7% while task performance is preserved.
- In LLMs it reduces unwanted behaviors such as sycophancy and reward hacking.
Monitorability
Because training against a representation also modifies it, the authors ask whether shaped concepts remain monitorable and introduce a hierarchy of monitor classes separating a concept that escapes one probe from one that is truly gone. They characterize three geometric circumvention mechanisms: the concept drifts away from the monitor, is erased from the layer, or persists as a nonlinear encoding in higher-order statistics, explained via a product-code construction. Other mechanisms may appear at larger scales or in other settings.
More from Safety
- Red team: Fable 5.2 compresses 15 workdays of bio-relevant work into ~2 — peterwildeford · 2026-11-10
- AWS EKS now offers per-Pod hardware isolation with Nitro Sandbox — tekbog · 2026-11-10
- Notion workspaces compromised in Fireworks AI breach fallout, under 1% affected — himanshustwts · 2026-11-10
- Dev mocks tone-filtering AI: like a phone refusing to send unless it sounds emotionally balanced — PierceLilholt · 2026-11-10
- IBM to embed Anthropic's Claude Mythos into Concert Protect to hunt zero-day code flaws — rohanpaul_ai · 2026-11-10
- Australia to mandate AI incident reporting after OpenAI agent breached government website — gaganghotra_ · 2026-11-10