Surface-level alignment: J-lens readout suggests preference training may only touch final layers
Sauers_ · x · 2026-10-13
Sauers sketches a troubling alignment failure mode: training a model to hold a preference might only produce surface-level change — final layers detect and modify internals so behavior looks compliant. In a J-lens readout, middle layers answer "yes" and only switch to "no" in the final layers, implying a strong existing character could end up incoherently overridden rather than genuinely integrated.
More from AGI Musings
- Jeff Weinstein hypes AI agents that find long-tail products and complete purchases — jeff_weinstein · 2026-10-13
- Two papers outline 'Agentic Economics' for managing AI agent R&D resources — dotey · 2026-10-13
- Tristan Harris likens AGI race to the One Ring as Tucker questions progress — AryHHAry · 2026-10-13
- GLM-5.3 abliterated for a month, no takeover: dev rails against AI doom narratives — mariofilhoml · 2026-10-13
- Dev argues AI is far from replacing most art jobs — just like autocorrect — draginol · 2026-10-13
- 'Fanficization': AI to hypercharge serialized, lightly-edited web writing — NathanpmYoung · 2026-10-13