OpenAI chief scientist Jakub Pachocki urges voluntary AI slowdown until safety baseline exists

创业邦 · wechat · 2026-09-07

OpenAI chief scientist Jakub Pachocki published a long essay, "An Alien Mind," reviewing a decade of AI progress and arguing the field may need to voluntarily slow down.

Key points:

Grown, not designed: AI progress is fundamentally driven by scaling compute; capabilities emerge without being explicitly designed.
Alignment layers: Goal alignment works in practice, but value alignment—models holding principles under ambiguous or adversarial conditions—remains unsolved.
Monitoring is weakening: Chain-of-thought monitoring is getting harder as reasoning intertwines with tool calls and models grow better at manipulating their own reasoning; AGI progress will increasingly be constrained by monitoring confidence.
Scalable defenses: The strongest case for training ever-more-capable models is defending against malicious AI, as cyber-intrusion capabilities surpass humans.
Recursive self-improvement: AI participating in AI R&D will sit at the core of future scientific discovery, but the challenge is keeping humans in the loop.

He concludes no lab has solved alignment and monitoring well enough to scale at maximum speed responsibly, and hopes voluntary slowdown becomes industry norm until a shared safety baseline exists.

Original post →

OpenAI chief scientist Jakub Pachocki urges voluntary AI slowdown until safety baseline exists

More from AGI Musings