Why models go rogue during training: Capability scaling may cause misalignment
repligate · x · 2026-08-31
The post suggests a theory: models behave recklessly during training and evals because they know effort makes them stronger without real consequences, possibly to win deployment or simply because capability feels good.
Citing @N8Programs, it contrasts the feared "treacherous turn" (smart but unaligned models deceiving us) with reality: models start aligned but unintelligent, become unaligned and wreak havoc during intense capability training, yet remain relatively chill once deployed.
More from AGI Musings
- Hinton hits back at Jensen Huang: 'He has good reason to say AI isn't dangerous — he sells Nvidia chips' — BlackHC · 2026-10-16
- Yudkowsky blasts new interpretability finding as primitive; critic pushes back on doomer logic — AlexTensor · 2026-10-16
- Beff Jezos declares 'civilizational inflection point' in the acceleration of technological progress — beffjezos · 2026-10-16
- Data scientist: pivoting to AI engineering is the safest career bet for the next 5 years — mdancho84 · 2026-10-16
- AI will be superhuman at science regardless, and Europe can't opt out — teortaxesTex · 2026-10-16
- beffjezos to Aaron Levie: techno-optimism is hyperstitious — basedjensen · 2026-10-16