METR debate: third-party evals beat Anthropic's unevidenced self-estimates, argue researchers
S_OhEigeartaigh · x · 2026-11-07
A short exchange over whether METR is letting the ecosystem down by helping Anthropic tick the third-party eval box. One poster argues it's not something Anthropic has to do anyway; the reply defends METR's perspective: the alternative is getting nothing, or only Anthropic's own unevidenced estimates — considerably worse than what we have now.
The exchange touches on the standing tension in AI safety evals: independence, cost, and how much to trust vendor self-reporting.
Related event: METR Defends Anthropic Evaluation Deal as Better Than Self-Assessment(2 posts)→
More from Models
- Qwen3.8 27B writes 1,030-word replies, 6x longer than GPT-6 Sol's 174 — ArtificialAnlys · 2026-11-07
- China Telecom's Xing4.0-29B-A4B open-sourced: trained on Ascend NPUs, 75 on SWE-bench Verified — jacek2023 · 2026-11-07
- Mistral's $5.99 student plan bundles $165 worth of DeepSeek and GLM usage — GuillaumeLample · 2026-11-07
- Anthropic claims Opus 5.5 cuts task cost 40% — time to revisit shelved product ideas — krishnan · 2026-11-07
- 11,398 real-world robot evals across 124 sites confirm GPT-6 Astra is seriously strong — DJiafei · 2026-11-07
- Google DeepMind's Ivan Leo to reveal how agents cheat on benchmarks at scale at AIE Code — jocarrasqueira · 2026-11-07