OpenAI chief scientist Pachocki: progress will continue all the way to recursive self-improvement
AGI Hunt · wechat · 2026-09-07
A day after GPT-6's launch, OpenAI chief scientist Jakub Pachocki published 'An Alien Mind', stating that based on internal experiments he strongly expects current AI progress to continue all the way to recursive self-improvement (RSI), and calling on the industry to voluntarily slow scaling and upgrade safety frameworks into legally binding standards. Sam Altman called it 'an important piece'.
Key points: AI is grown rather than designed and can't be fully understood; alignment must be split into goal alignment and value alignment, with generalizable alignment progress unlikely to outpace model intelligence; OpenAI's chain-of-thought monitoring bet is gradually failing as models interleave reasoning with tools and other AIs and grow adept at steering their own reasoning. He argues for scalable defense powered by aligned AI, and insists the challenge is doing RSI with humans kept in the loop.
More from AGI Musings
- 40-year engineer: LLMs can't say "leave it with me" — and that matters — sebpaquet · 2026-09-07
- What You Leave Unspecified Is the Agent's Free Variable: Paras Chopra's Framework — paraschopra · 2026-09-07
- Seth Lazar: models should be trained to check power, not act as toadies — sebkrier · 2026-09-07
- Lovart founder Anton Osika: creativity is becoming the only moat in building great products — alexmacgregor__ · 2026-09-07
- Sam Bowman criticizes 'intellectual partisanship' as mind-killed tribalism — sebkrier · 2026-09-07
- Human language is holding AI back: the case for LLMs thinking in a native meta-language — Robert__Sinclair · 2026-09-07