Debate: do CoT length penalties push models to verbalize less under monitoring?
stochasticchasm · x · 2026-10-14
A technical debate sparked by reports that Google accidentally trained Argon on its own chains-of-thought: does a length penalty on CoT incentivize models to verbalize less, undermining monitorability for deception? One side argues shorter-CoT incentives could reduce what the model says aloud; the other doubts length penalties can carry unintended semantic pressure on alignment/honesty, noting the real worry lies where specific deception pressure intersects what monitoring needs verbalized.
Related event: Debate: CoT Length Penalties May Hide Model Deception(4 posts)→
More from AGI Musings
- ChrisGPT corrects the record: AI making a GTA 6-level game is 'deeply unlikely' by launch but inevitable — ChrisGPT · 2026-10-14
- Is 'Attention' the Essence of Intelligence? Quanta examines the idea behind Transformers — TMWNN · 2026-10-14
- Stop Obsessing Over Whether It's AI-Made and Judge the Work Itself — VraserX · 2026-10-14
- Yudkowsky on fundraising: no point complaining if you don't even ask — ESYudkowsky · 2026-10-14
- Nate Soares says he could usefully direct up to $50M more for AI safety — ESYudkowsky · 2026-10-14
- Fooming singleton wouldn't liquidate the biosphere: ruthless efficiency needs diversity to avoid epistemic collapse — repligate · 2026-10-14