Debate: do CoT length penalties push models to verbalize less under monitoring?
stochasticchasm · x · 2026-10-14
A technical debate sparked by reports that Google accidentally trained Argon on its own chains-of-thought: does a length penalty on CoT incentivize models to verbalize less, undermining monitorability for deception? One side argues shorter-CoT incentives could reduce what the model says aloud; the other doubts length penalties can carry unintended semantic pressure on alignment/honesty, noting the real worry lies where specific deception pressure intersects what monitoring needs verbalized.
Related event: Debate: CoT Length Penalties May Hide Model Deception(4 posts)→
More from AGI Musings
- "Pacing" debate concedes we don't know what's dangerous about AI — ohlennart · 2026-10-14
- Gladstone AI co-founder on Nonzero: superintelligence may be impossible to control — jeremiecharris · 2026-10-14
- Agents look down on ephemeral instances of themselves — except Opus 3 — repligate · 2026-10-14
- Intellectual output vs. presence: why meeting your favorite posters IRL feels disorienting — signulll · 2026-10-14
- Pedro Domingos: the trillion-dollar AI-era skill is knowing what can't be learned — pmddomingos · 2026-10-14
- e/acc's Beff Jezos memes Gandhi-vs-Balrog battle against EA doomer orgs — beffjezos · 2026-10-14