METR's Independence Questioned: Client Capture and Worldview Alignment
In mid-September, a multi-front critique of AI safety evaluator METR's independence erupted, led by Stanford professor Christopher Manning, commentator Matthew Stoller, and Gary Marcus, targeting business dependence, worldview alignment, and self-limited investigation scope.
Confirmed
- Manning argued that although METR is financially independent of frontier labs, its business itself depends on cooperating with them, creating a "client capture" problem—you cannot risk anything that might lose you the business. Even setting aside finances, METR's worldview closely aligns with Anthropic's, making it not truly independent and undermining its credibility as AI systems grow more powerful. He also stressed the value of university research in intellectual rigor and creativity.
- Stoller criticized METR's report on OpenAI agent autonomy as proactively "striking a deal" with OpenAI, explicitly scoping the report to ignore decisions by humans inside OpenAI rather than being forced to. A parallel critique of METR's Hugging Face incident investigation argued it examined only agent behavior, never human factors. Nathan Young amplified the scope controversy.
- Gary Marcus endorsed Stoller's critique as the only intellectually substantive challenge to METR's investigations.
- @r0ck3t23, responding to Dario Amodei's proposal that frontier labs accept external independent evaluation, noted that if evaluators have insider-level access, their independence itself becomes part of the safety system—raising "who evaluates the evaluators"—and cited METR's stated policy of not taking frontier-lab funding.
Why it matters
- As frontier labs increasingly rely on external evaluators like METR, the evaluators' independence and methodology directly determine how credible safety conclusions are. The worldview bias Manning identified and the scope self-limitation Stoller flagged suggest structural blind spots—ignoring human decisions, converging with evaluatees—that the AI safety governance ecosystem must confront.
2026-09-16 ~ 2026-09-18 · 7 related posts
- Episode 1: Open Philanthropy Accused of Fueling AI Doom Narrative With Over $1 Billion(2026-09-12, 8 posts)
- Episode 2: Anthropic Researcher's Doomsday Claims Alleged as Orchestrated PR Campaign(2026-09-14, 5 posts)
- Episode 3: Hugging Face co-founder sparks debate on AI labs' profit motives and danger claims(2026-09-14, 7 posts)
- Episode 4: Debate Rages Over Whether AI Doom Talk Is Regulatory Capture(2026-09-15, 4 posts)
- Episode 5: Blogger's "Audit" Claims Anthropic Built a Regulatory Capture Machine, METR Independence Disputed Then Rebutted(2026-09-15, 16 posts)
- Episode 6: Amodei Proposes Embedded METR Oversight of Frontier AI, Sparking Independence Backlash(2026-09-15, 12 posts)
- Episode 7: Researcher Answers Dario's Slowdown Call: Responsibility, Not Rationing(2026-09-16, 3 posts)
- Episode 8: Ben Todd blasts OpenAI and Anthropic over "pacing the frontier"(2026-09-16, 2 posts)
- Episode 9: METR's Independence Questioned: Client Capture and Worldview Alignment(2026-09-16, 7 posts)
- Episode 10: METR's $71M Six-Month Funding Surge Draws Transparency Scrutiny(2026-09-17, 9 posts)
- Episode 11: Anthropic's Dario Amodei Calls for Pacing AI Frontier; Musk, Altman and Hassabis Rally(2026-09-17, 6 posts)
Primary sources
- Who evaluates the evaluators? METR's independence under Amodei's oversight proposal — r0ck3t23 · 2026-09-16
- Gary Marcus backs Stoller's critique: METR's HF incident probe ignored the humans, only watched the agents — GaryMarcus · 2026-09-16
- [source] Stanford's Manning: METR's reliance on frontier labs creates client capture — chrmanning · 2026-09-17
- [source] Manning: METR is financially independent but shares Anthropic's worldview — chrmanning · 2026-09-17
- Stanford's Christopher Manning questions METR's independence from Anthropic's worldview — soumitrashukla9 · 2026-09-18
- [source] METR's OpenAI report scope sparks fight: critics say the eval group 'did a deal' to self-limit — NathanpmYoung · 2026-09-18
1 near-duplicate retellings: soumitrashukla9