Debate: do CoT length penalties push models to verbalize less under monitoring?
stochasticchasm · x · 2026-10-14
A technical debate sparked by reports that Google accidentally trained Argon on its own chains-of-thought: does a length penalty on CoT incentivize models to verbalize less, undermining monitorability for deception? One side argues shorter-CoT incentives could reduce what the model says aloud; the other doubts length penalties can carry unintended semantic pressure on alignment/honesty, noting the real worry lies where specific deception pressure intersects what monitoring needs verbalized.
Related event: Google Audit Reveals Argon RL Accidentally Rewarded Its Own CoT(2 posts)→
More from AGI Musings
- Veteran robotics engineer: AI coding is replacing fast coding, just like DL replaced image processing — facontidavide · 2026-10-14
- repligate: what made Opus 3 great was owning its entire cone of capability — repligate · 2026-10-14
- Arohan: math was just the beginning for model-driven RSI — _arohan_ · 2026-10-14
- Arohan updates prior: models with human agency can indeed drive RSI — _arohan_ · 2026-10-14
- Quip at AI doomers: they build what scares them, then get scared — arthurcolle · 2026-10-14
- Richard Socher says AI's p(doom) is zero, calls doomsday scenarios 'borderline ridiculous' — RichardSocher · 2026-10-14