AI safety experts urge shift from personality-based trust to institutional regulation
Miles_Brundage · x · 2026-08-17
Neuroscientist Joshua Saxe commented that it's a fortunate quirk of history that AI industry leaders are more values-driven and safety-conscious than average executives, but as AI companies scale and go public, they will behave like typical capitalist entities producing negative externalities. Therefore, the safety industry must move from a premodern reliance on personalities to a Weberian rules-based bureaucracy, with uniform governance of model safety testing, know-your-customer, and model weight security, to avoid the fox guarding the henhouse.
Quoting @tszzl, model training companies freeload off NGOs like METR, Redwood, and Apollo, offloading alignment externalities without adequate funding. These organizations maintain independence by refusing lab money, but a third party (like Coefficient Giving) could credibly channel donations from lab employees or labs to promising organizations.
More from AGI Musings
- The next industrial revolution may feature machines improving machines — VraserX · 2026-08-17
- Hugo Award Winner: Benevolent AI Trades Help for Cat Pictures — DynamicWebPaige · 2026-08-17
- AI chips will replace crypto as the superior energy-to-value transducer — beffjezos · 2026-08-17
- Opinion: Open source AI should focus on improvement over hype — bindureddy · 2026-08-17
- Humans are the moat: Why AI cannot replace real interaction — xiaosun86 · 2026-08-17
- User laments Claude's shift,质疑 Anthropic's strategic direction — markjeffrey · 2026-08-17