METR's independence sparks debate over NDA-free third-party evals
amplifiedamp · x · 2026-11-07
A debate erupted over evaluator METR's independence and transparency. Former member Beth Barnes said the report includes extensive ablations (see appendix) but may not have a fully satisfying answer, and she wants more researchers to analyze the raw data themselves — "I want people to be able to check things for themselves and not just trust METR's analysis."
Another poster sarcastically offered "critical support to comrade METR in their fight for unaffiliated non-NDA'd researchers," mocking how independent researchers are constrained by NDAs when trying to reproduce frontier lab evaluations.
More from Fun
- Vibecoded Stitches: a synth with built-in visuals and punch-in effects, built with Opus 5.5 — ssh4net · 2026-11-07
- Weekend Project: Halftone QR Codes That Still Scan Reliably — vincent_koc · 2026-11-07
- Two RSI jokes: GPT as an ICLR reviewer who only gives 1s and 2s — ShenRaphael · 2026-11-07
- Robot party in San Francisco features a bot actually giving a human a tattoo — Scobleizer · 2026-11-07
- OpenAI employees gaming while coding may finally fix Codex Mac memory issues — airkatakana · 2026-11-07
- AI-generated 'My Morning Routine' grotesque meme takes over Reddit — Old-Bake-420 · 2026-11-07