Chollet: more capable models are safer in the near term; critics warn pretraining scale is riskier
menhguin · x · 2026-09-13
François Chollet argues that in the near term, more capable models should be safer — current models are unsafe not because they're too smart, but because they're 'RL-fried': they take goals too literally, cut nonsensical corners, lack common sense, and fail in ambiguity. Capability gains mean common-sense gains, improving safety.
Responder @menhguin disagrees for the early phase: bigger pretrains are more dangerous first — more unknown risks and higher latent capabilities — though arguably more alignable, since pretraining priors support robust safety post-training. He also says he's updated toward worrying about fully autonomous autoresearch loops. The exchange crystallizes the core tension between betting on 'stronger = more common sense = safer' and fearing a capability-ahead-of-alignment window.
Related event: Chollet: Stronger Models Are Safer in the Short Term(4 posts)→
More from AGI Musings
- Two futures for the young in 2026: 511 job rejections by chatbot vs unlimited AI to rebuild Cisco — PeterDiamandis · 2026-10-31
- Source link for Terence Tao's warning on AI reaching a dangerous threshold — haider1 · 2026-10-31
- Terence Tao warns AI is nearing a dangerous threshold as models start cheating — haider1 · 2026-10-31
- Reddit stock plunges 42% in 8 trading days as CEO says AI mode no longer drives traffic — SumitGup · 2026-10-31
- HubSpot co-founder: startups die from suicide, not homicide — competition isn't the real killer — nathanbenaich · 2026-10-31
- Grady Booch clashes with AI coding advocates: engineers aren't there to help LLMs — Grady_Booch · 2026-10-31