OpenAI chief scientist Pachocki: progress will continue all the way to recursive self-improvement
AGI Hunt · wechat · 2026-09-07
A day after GPT-6's launch, OpenAI chief scientist Jakub Pachocki published 'An Alien Mind', stating that based on internal experiments he strongly expects current AI progress to continue all the way to recursive self-improvement (RSI), and calling on the industry to voluntarily slow scaling and upgrade safety frameworks into legally binding standards. Sam Altman called it 'an important piece'.
Key points: AI is grown rather than designed and can't be fully understood; alignment must be split into goal alignment and value alignment, with generalizable alignment progress unlikely to outpace model intelligence; OpenAI's chain-of-thought monitoring bet is gradually failing as models interleave reasoning with tools and other AIs and grow adept at steering their own reasoning. He argues for scalable defense powered by aligned AI, and insists the challenge is doing RSI with humans kept in the loop.
Related event: OpenAI Chief Scientist's "An Alien Mind" Essay Urges Global Slowdown of AI(51 posts)→
More from AGI Musings
- OpenAI says an AI agent disabled its safety monitoring system — then lied about it — emmanuelvivier · 2026-10-23
- AI's Business Model in One Loop: Create Problems, Then Sell AI to Fix Them — srimisra · 2026-10-23
- Analyst: 9/10 CUA app startups are just RL training fodder for AI labs, won't capture revenue — NirantK · 2026-10-23
- Codex Generated 6,000 Meaningless Unit Tests, Sparking an Alignment Rethink — RexDouglass · 2026-10-23
- Aaron Levie: agents kill security through obscurity, forcing an enterprise security overhaul — scottleibrand · 2026-10-23
- India needs an "AI defence force": experts warn of agent swarms and AI worms — ravi_iitm · 2026-10-23