100+ Experts Sign Letter Demanding Five Minimum Standards for Credible Embedded AI Evaluations
random_walker · x · 2026-09-18
A public letter signed by 100+ researchers, including Arvind Narayanan (randomwalker), calls on frontier AI companies to make embedded third-party evaluations credible. While welcoming companies' openness to embedded evaluators for assessing escalating AI capabilities and risks, the letter argues such evaluations must have scientific objectivity, transparency, independence, and robust protections against interference.
The letter lays out five minimum requirements, centering on:
- Evaluator independence from the companies being evaluated
- Protection from retaliation
- Real access to systems, real-world harm incidents, and companies' training, deployment, oversight, operational, and safeguard practices
The authors warn that without these guarantees, embedded evaluations risk becoming a formality as AI risks escalate.
More from AGI Musings
- Naval amplifies sharp take: sincere AI doom believers are more dangerous than cynics — naval · 2026-09-19
- npm co-founder Laurie Voss: agents are eating SDLC, product engineering pays $240k — round · 2026-09-19
- Yesterday's rebels become today's orthodoxy: a pattern that keeps repeating in AI — soumitrashukla9 · 2026-09-19
- Memory vs consciousness: why amnesiacs aren't a true counterexample — yeastsplainer · 2026-09-19
- Full CNBC Video: Suleyman Questions Anthropic's Constitution on AI Welfare — rohanpaul_ai · 2026-09-19
- NumPy Creator Travis Oliphant: Renting Intelligence Builds Landlords, Owning Models Builds Ecosystems — teoliphant · 2026-09-19