Zvi on Jakub Pachocki's "An Alien Mind": recursive self-improvement is coming and CoT monitoring is fading
Don't Worry About the Vase (Zvi) · rss · 2026-09-08
Zvi Mowshowitz breaks down OpenAI Chief Scientist Jakub Pachocki's essay "An Alien Mind," calling it a rare, unvarnished warning.
Core positions:
- Based on internal results since the 2023 "RLSlow" project, Pachocki expects progress to sustain into recursive self-improvement (RSI) — machines meaningfully smarter than humans within our lifetime
- No one is prepared; OpenAI will unilaterally withhold further scaling as needed, but broader interventions including international coordination are required
- AI doesn't need to exceed humans on every axis to transform the world or become dangerous — just enough axes
Alignment views:
- Alignment is the core problem, split into goal alignment vs. value alignment; the fundamental challenge is generalization of values
- Chain of Thought monitoring, heavily invested in, is progressively losing effectiveness
- The remaining argument for scaling is cyber defense against scaled AIs
Zvi adds: competitive pressure pushes toward RSI over math/medical research, and Pachocki is ultimately betting on "automated alignment researchers" — with both acceleration of alignment work and slowing capabilities needed.
More from AGI Musings
- RCT: ChatGPT temporarily boosts consultants' data science skills, gains vanish without it — EricTopol · 2026-10-26
- Dev on AI games backlash: reject AI slop, not the tool itself — ezshine · 2026-10-26
- Worry less about AI than about the bad actors wielding it — zakkohane · 2026-10-26
- If an agent can rewrite itself, does your consent still mean anything? — tallmetommy · 2026-10-26
- Voice AI agents as the stepping stone from Palm Pilot to iPhone — curious_vii · 2026-10-26
- 109 of 529 AI Agent Incidents Involved No Adversary, Arguing Red Teams Must Test Normal Operation — anacondainc · 2026-10-26