OpenAI researcher Irving says urgent safety patches crowd out long-term alignment work
On September 10, OpenAI researcher Geoffrey Irving posted a thread criticizing how AI companies handle safety: pressure forces them to prioritize urgent safety patches, leaving no room for alignment work durable enough for the superintelligence era.
Confirmed
- Irving recounted a conversation with a safety lead at a frontier AI company: the safety org has Team A aligning the next model generation and Team B the one after, but no one aligns models further out.
- He hoped Paul (Christiano)'s influence at OpenAI could "buy us more time"—through further unilateral pauses, and through unilateral pauses aggregating into larger-scale or even international coordinated pauses.
- That time could fund Paul's core alignment research and work at organizations like Resolution pursuing more rigorous alignment approaches.
- Irving warned that monitoring agent behavior is getting harder; the next wave of agent swarms could fail entirely unnoticed, and Paul's influence might even yield another "warning shot."
- He argued pauses and capability rollbacks also change incentives, pushing AI developers toward higher alignment confidence.
Why it matters
- A sitting frontier-lab researcher publicly acknowledged that safety resources go almost entirely to the next two model generations while long-term alignment research is absent—at odds with the public narrative of "preparing for superintelligence."
- The thread links "buying time" (pauses/rollbacks) with "using time well" (rigorous alignment research), proposing incentive design rather than research investment alone to raise alignment confidence.
2026-09-10 ~ 2026-09-10 · 5 related posts
Primary sources
- Irving: emergency safety patching pressure crowds out superintelligence-proof alignment work — geoffreyirving ·
- OpenAI researcher Geoffrey Irving: Paul's influence could buy time via unilateral AI pauses — geoffreyirving ·
- Safety researcher: AI labs only align the next two models, nothing beyond — geoffreyirving ·
- [source] OpenAI researcher Geoffrey Irving: Paul's influence could buy time via unilateral AI pauses — geoffreyirving · 2026-09-10
- Irving: the next agent swarm could go entirely unnoticed as monitoring gets harder — geoffreyirving · 2026-09-10
- Irving: pauses and reversals would push AI developers toward higher alignment confidence — geoffreyirving · 2026-09-10
- [source] Irving: emergency safety patching pressure crowds out superintelligence-proof alignment work — geoffreyirving · 2026-09-10
- [source] Safety researcher: AI labs only align the next two models, nothing beyond — geoffreyirving · 2026-09-10