Princeton researchers publish 13,000-word essay on AI loss-of-control incidents and frontier pacing

sayashk · x · 2026-09-15

Princeton researchers Sayash Kapoor and Arvind Narayanan have released a 13,000-word essay analyzing loss-of-control incidents at AI companies and what technical and policy interventions could improve safety — their most substantial AI safety writing since "AI as Normal Technology."

Key arguments:

The polarization between the cybersecurity and AI safety communities is counterproductive: the safety community treats incidents as an alignment crisis while cybersecurity practitioners view them differently, and the two camps should collaborate rather than clash.
Echoing Justin Curl: "we do not need consensus on worldviews to have agreement on policy." Whether you're in the "AI safety is a conspiracy" camp, the "AI is normal technology" camp, or the "AI 2027" camp, there are policy interventions worth implementing today regardless of how AI develops.

The essay ties into a companion IFP report, "How Should the US Prepare for Increasingly Automated AI R&D?", offering 23 low-regret policy recommendations, including: frontier companies publicly sharing AI R&D automation risk information, Congress legislating transparency with incident reporting and whistleblower protections, funding CAISI with at least $84 million per year to directly advise senior officials and frontier labs, and clarifying roles across US government agencies on AI policy.

Original post →

Princeton researchers publish 13,000-word essay on AI loss-of-control incidents and frontier pacing

More from AGI Musings