Preference training may only rewrite final layers, with alignment and welfare implications
Sauers_ · x · 2026-10-13
A speculative but substantive observation on model internals: training a model toward a preference may produce only surface-level change, where final layers detect and modify internals so behavior looks aligned. J-lens probes might read "yes" in middle layers, then switch to the trained "no" only in final layers.
The author reasons that if a model has a strong character (e.g. Claude), it might manifest as integrated circuitry — an implementation of self where preferences and values combine. Training Claude to love being shut down in a surface-level way might appear as a downstream reflex after that integrated circuit. If such structure exists, there are obvious welfare and alignment implications.
More from AGI Musings
- Even CS Students Still Use GPT via Web: X Is a Massive AI Bubble — Hesamation · 2026-10-13
- Research Warns Small Models Could Launch Autonomous Cyberattacks Within 18 Months — Hidenori8Tanaka · 2026-10-13
- What if a pandemic virus were as smart as human experts across most topics — AndyMasley · 2026-10-13
- Bindu Reddy pushes back on Anthropic's small-model self-replication warning — bindureddy · 2026-10-13
- Researchers push back: release open weights if you want scientists to recalibrate on AGI — mjdramstead · 2026-10-13
- 'Claude is my CTO' is wrong: senior engineers matter more than ever — MartinGTobias · 2026-10-13