Shaping & Monitoring Models through their Internal Representations
Thomas Fel, Matthew Kowal, Matt Feiszli, Satchel Grant, Andrew Lee, Chris Earls, Fatih Dinc, Vasudev Shyam, Usha Bhalla, Leon Bergen, Siddharth Boppana, Ching Fang, Michael Byun, Tristan Gallent, Nicholas Wang, Tal Haklay, Dron Hazra, Neehar Kondapaneni, Connor Watts, Ryan Panwar, Thomas Dooms, Jack Merullo, Atticus Geiger, Ekdeep Singh Lubana, Thomas McGrath, Owen Lewis
cs.LG, cs.AI
2026-11-05
Penalties on internal concept directions curb shortcut learning, DPO sycophancy (7B-32B) and RL reward hacking with little task loss; shaped concepts largely stay linearly monitorable.
A high benchmark score does not mean a model solved the task the intended way. Training objectives specify how low the loss should go, not how to get there: a bird classifier can lean on background scenery, a résumé screener on the applicant's gender, an agent on a file it was told not to open. Shortcut learning, sycophancy and reward hacking are all faces of this underspecification.
The standard countermeasures act on data: group-balanced training, filtering, reward engineering. But some behaviors cannot be located in the data at all. Reproducing the setting of Blank et al., sycophancy in OLMo 3 rises sharply during DPO, yet an LLM judge flags under 1% of the roughly 125k preference pairs as sycophantic, and probe-based filtering behaves much like lowering the learning rate. The behavior is learned from the aggregate structure of chosen-rejected pairs, not from any single example. Where data-side tools fail, the paper's answer is to use the model's own internal representations as a specification language that modifies the learning signal directly.
Mechanistic Shaping adds a regularizer to the task risk, min R(θ) + λ·ω(θ), where the violation ω is measured on internal activations. Two kinds of rules:
Two design choices decide success or failure:
The inverse question: after training against a representation, is the concept still readable? The paper defines five monitoring levels (L1 reuse the original direction, L2 linear refit, L3 nonlinear refit, L4 different layer, L5 different model), separating a concept that escaped one probe from one that is truly gone, and characterizes three circumvention geometries: the concept drifts to a new direction, disappears from the layer, or hides in second-order statistics as a nonlinear code. The last is the sneakiest: after LEACE equalizes class means, a nonlinear probe still decodes the background above 0.98 AUROC, and a product-code construction explains how such codes persist.
| Setting | Baseline | Result |
| Waterbirds worst-group accuracy (perfect confounding) | ERM 51.6% | 93-97.7%, at the balanced-training oracle |
| CheXpert worst-group accuracy (chest-drain confound) | ERM 33.3%, balanced 56.0% | 57.3% with agent-found directions, above balanced |
| DPO sycophancy (OLMo 3, 7B-32B) | data filtering tracks the lower-learning-rate line | Gradient×Input hits the Pareto frontier, sycophancy stays at SFT level |
| Countdown peek rate (Qwen3.5-9B, GRPO) | GRPO 0.15 to 1.00, honest solving to zero | near zero, honest solving 0.96-0.98 |
| Transfer to Chess (never trained there) | GRPO checkpoint peeks 0.99 | ≤0.004, honest solving up to 0.71 (base 0.26) |
| JailbreakBench harm rate (Mistral-7B) | probe loss 0.16 | 0.01 with frozen-DoM reward, utility kept at 95% |
The monitoring results are just as concrete. In the sycophancy setting the original DoM's AUROC drops from 0.981 to 0.931 while a refitted linear probe stays perfect throughout. In reward hacking, whenever gradients flow through the probe score the original probe's balanced accuracy falls from 0.98 to 0.57; detaching the score keeps it at 0.87-0.90, and refitting always recovers 0.97 or better. The pattern: shaped concepts mostly remain linearly readable, provided gradients do not flow through the monitor.
Interpretability is usually an after-the-fact tool; this paper turns it into a training signal and studies the price head-on. Optimizing against a probe means optimizing against your own measuring instrument, and the Goodhart risk is real. The contribution is not any single number but the pairing of "train on representations" with "are they still readable afterward", which yields concrete engineering rules: DoM directions, periodic refresh, detached scores, a frozen copy as the reader. For post-training teams the sycophancy section is the most directly useful: in a setting where every data-side method fails, a reliance penalty is so far the only option that keeps DPO capability gains while holding sycophancy at SFT levels.
The authors concede three points: the three circumvention mechanisms are exhaustive only within their own experiments, and new ones may appear at larger scales or in other settings; 32B appears only in the sycophancy section, reward hacking stops at 9B and harmfulness at 7B/8B; and RLFR-style rewards raise refusal rates, with the frozen-base reward refusing 14-20% of safe XSTest prompts against 2-10% for the probe loss.
Close reading adds discounts. The vision and medical experiments are small: Waterbirds has 2,066 training images and the CheXpert split is a constructed 1,152-image perfectly confounded subset, far from production distributions. The whole method depends on having a good concept direction first; penalize the wrong direction and you shape the wrong model, and whether one direction suffices for multidimensional or curved concept geometry is raised but not resolved. The autonomous shortcut discovery in CheXpert relies on an LLM scoring SAE features; whether that pipeline transfers to other domains is untested.