AI safety experts urge shift from personality-based trust to institutional regulation
Miles_Brundage · x · 2026-08-17
Neuroscientist Joshua Saxe commented that it's a fortunate quirk of history that AI industry leaders are more values-driven and safety-conscious than average executives, but as AI companies scale and go public, they will behave like typical capitalist entities producing negative externalities. Therefore, the safety industry must move from a premodern reliance on personalities to a Weberian rules-based bureaucracy, with uniform governance of model safety testing, know-your-customer, and model weight security, to avoid the fox guarding the henhouse.
Quoting @tszzl, model training companies freeload off NGOs like METR, Redwood, and Apollo, offloading alignment externalities without adequate funding. These organizations maintain independence by refusing lab money, but a third party (like Coefficient Giving) could credibly channel donations from lab employees or labs to promising organizations.
More from AGI Musings
- Shogun 2 remaster review-bombed over AI-upscaled graphics, sparking dev debate — draginol · 2026-10-16
- Pedro Domingos: Everyone must have their own unique model, anything else is dangerous — pmddomingos · 2026-10-16
- Interp researcher pushes back: if you can't simulate a neuron, you haven't explained it — nabla_theta · 2026-10-16
- Virtue ethics meets AI alignment: what would it mean to make a model virtuous? — joelbot3000 · 2026-10-16
- Professor: this rec letter season I'm evaluating the student plus Claude — RoozbehMottaghi · 2026-10-16
- Prediction: Riemann Hypothesis likely falls to AI before 2030 — gautamcgoel · 2026-10-16