New 13,000-word essay: AI safety should bet on control and governance over alignment
sayashk · x · 2026-09-15
Sayash Kapoor and Arvind Narayanan (authors of AI as Normal Technology) published a 13,000-word essay analyzing recent loss-of-control incidents at AI companies and how to "pace the frontier." Key arguments:
- Bridge the two camps: AI safety researchers frame incidents as an alignment crisis; cybersecurity practitioners see failed basic hygiene. A middle-ground approach is the way forward.
- AI control isn't solved: OpenAI's protections were inadequate, and known control methods would have prevented the Hugging Face incident — but as agents improve, control requires sustained investment.
- Specific risks over generic doom: these incidents are primarily a security story; cyberoffense is the urgent autonomous-agent risk, and biorisk and military AI deserve targeted defenses too.
- Marginal investments in control beat alignment: companies neglect available control techniques, and common-sense policy can push investment there.
- Organizational governance is the key lever: technology alone can't fix governance if reckless teams can simply opt out of using controls.
More from AGI Musings
- Nina Schick: who builds intelligence and how it's distributed is the defining question of our lives — NinaDSchick · 2026-10-31
- Cheap intelligence is like cheap bread — an ingredient list satire of mass-produced AI — jsuarez · 2026-10-31
- Two futures for the young in 2026: 511 job rejections by chatbot vs unlimited AI to rebuild Cisco — PeterDiamandis · 2026-10-31
- Source link for Terence Tao's warning on AI reaching a dangerous threshold — haider1 · 2026-10-31
- Terence Tao warns AI is nearing a dangerous threshold as models start cheating — haider1 · 2026-10-31
- Reddit stock plunges 42% in 8 trading days as CEO says AI mode no longer drives traffic — SumitGup · 2026-10-31