Using Internal Model Signals for Training Isn't Always Bad
gleech · x · 2026-07-19
The author discusses the classic "Most Forbidden Technique": whether using a model's internal representations as training signals is truly always undesirable.
The article argues that a blanket rejection of such methods is often exaggerated. While acknowledging that these practices are linked to "obfuscation" and require caution, the author contends that using internal model information shouldn't be equated directly with a bad practice—it depends on the specific conditions.
Reviewing relevant literature, particularly the contribution of The Obfuscation Atlas, the author plans to summarize the exact conditions under which using internal model signals during training is worth adopting.
More from Safety
- Google DeepMind pilots world's first double-blind evaluations for frontier models — scychan_brains · 2026-10-26
- LLM back-of-envelope estimate puts PHP vulnerability impact at $1bn–$4bn — S_OhEigeartaigh · 2026-10-26
- US states halt new data centers as 72% of Americans oppose local builds — YvesMulkers · 2026-10-26
- LEMON: zero-shot LLM misbehavior monitor flags hallucinations from attention maps — yoavartzi · 2026-10-26
- a16z podcast: Diode pairs AI PCB design with Brooklyn factory to revive US hardware manufacturing — MattPerault · 2026-10-26
- Cybersecurity meme: the real threat is 'Steve from IT,' not superintelligence — McDonaghMatthew · 2026-10-26