AI safety researchers argue even finance-grade governance may not be enough for AI
geoffreyirving · x · 2026-10-23
Geoffrey Irving started a thread arguing that while much AI-risk discussion focuses on closing the gap between AI labs' methods and rigorous industries like aviation, nuclear power, and AI hardware, it is unclear that reaching that level would be sufficient.
Quoting him, Beba Cibralic — who worked in AI in finance — adds that many serious governance proposals borrow from finance: independent audits, bank-style supervision, or an AI version of FINRA. These beat the status quo at AI companies, but shouldn't be treated as sufficient:
- Finance has had decades to build such controls and still suffers major failures, plus many small ones never make headlines, making these mechanisms look more reliable than they are;
- Financial risks are well understood and predictable, unlike the risks posed by AI systems.
Takeaway: lifting AI to engineering-grade rigor is worth doing, but AI governance may need to go beyond existing industry templates.
More from AGI Musings
- Senior researcher rebuts 'one-person pretraining' fear: your fundamentals matter more, not less — RexDouglass · 2026-10-23
- NBER releases 'The Economics of Transformative AI' edited by Agrawal and Brynjolfsson — JMateosGarcia · 2026-10-23
- Pedro Domingos: AI Is an Intelligence Lever — the Smarter You Are, the Smarter It Makes You — pmddomingos · 2026-10-23
- Anil Seth: Consciousness Is a Property of Life, Not Computation — Building Conscious AI Is a Bad Idea — anilkseth · 2026-10-23
- Pedro Domingos predicts 2027 will see the most security vulnerabilities fixed in history — pmddomingos · 2026-10-23
- AI+Lean theorem proving loses to Bitter Lesson: OpenAI math gains come from non-formalized reasoning — Singularitarian · 2026-10-23