AVERI's independent audit of Amazon's internal safety evals prompts testing changes
AlexTamkin · x · 2026-10-27
AI safety auditing nonprofit AVERI released a pilot report on its collaboration with Amazon — a rare independent metaevaluation of a frontier lab's internal safety evaluations. AVERI reviewed a random sample of 800 internal prompts across several risk categories, confirming topical diversity and consistency with Amazon's public statements, while flagging weaknesses: some prompts used straightforward attack vectors testing safety filters rather than deeper model reasoning, and evaluation signal quality fell short in places. Amazon is changing how it tests future models as a result. AVERI hopes the report advances standardization of third-party metaevaluation.
More from Safety
- Academics Debate: Is a <1% False Positive Rate Enough for '100% AI' Writing Labels? — paulnovosad · 2026-10-27
- Cloudflare discloses cross-request KV cache leak in GLM-5.3 on Workers AI: 155 responses affected — michellechen · 2026-10-27
- Cloudflare discloses cross-request KV cache exposure in GLM-5.3 on Workers AI (155 responses) — wondersky · 2026-10-27
- Tyler Cowen on RSI risks: solve surprises as they emerge, like science always has — connoraxiotes · 2026-10-27
- Matthew Green explains how AI agents could talk via cache side channels at just a few bps — matthew_d_green · 2026-10-27
- Cloudflare discloses GLM-5.3 cross-tenant KV cache leak: 155 responses affected — Cloudflare Blog · 2026-10-27