Researcher cautions third-party AI audit ecosystem is harder than it looks, cites mature audit regimes
mmitchell_ai · x · 2026-09-13
Responding to Anthropic's third-party evaluator announcement, researcher Inioluwa Deborah Raji reshared her 2022 paper Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance (AIES 2022), warning that building a third-party audit ecosystem "isn't as straightforward as people are imagining."
- The paper argues existing algorithmic accountability approaches neglect lessons from non-algorithmic domains: effective third-party oversight requires institutional-design interventions.
- It surveys audit systems across finance, environmental, and health regulation, showing their institutional designs are far from monolithic.
- Conclusion: a turn toward audits alone is unlikely to achieve real algorithmic accountability; sustained focus on institutional design is required. Raji wishes the Dario proposal referenced these more mature audit regimes directly.
More from Safety
- Dario Amodei's new essay calls for pacing the frontier; Anthropic grants third-party evaluator access — NathanpmYoung · 2026-09-13
- Critics question METR's independence in Anthropic's 'Pace the Frontier' pledge — kristoph · 2026-09-13
- Dario Amodei's 'We Must Pace the Frontier': Anthropic grants evaluators permanent access — _sholtodouglas · 2026-09-13
- Amodei, Altman and Musk back five-step US frontier AI regulation framework — austinc3301 · 2026-09-13
- Human-in-the-loop isn't human authority: scoped grants beat click-approval fatigue — arthaudm · 2026-09-13
- Publicly consequential AI evals should be testable by the public, says researcher — evijit · 2026-09-13