METR gains more eval access, but critics worry about an evaluator monoculture
dioscuri · x · 2026-09-13
Responding to the discussion around METR getting more model access and third-party independent evaluators being consulted, the author calls it a good thing but worries that the evaluators come from a very tight monoculture — arguing the safety eval ecosystem needs more diversity rather than a small, like-minded club. Delivered in a jokey all-caps aside.
More from Safety
- Tyler Alterman on the hacking panic: more Y2K than COVID, use 1Password and update software — jd_pressman · 2026-10-31
- Inference-only chip proposal would bar RL rollouts for models in training — wfithian · 2026-10-31
- Every Used Codex and Agents to Survive the Railway Credential Breach — every · 2026-10-31
- Researchers debate whether more inference compute can substitute for training compute — teortaxesTex · 2026-10-31
- OpenAI blocks weapons details for teens, adds auto-ending chats fixated on violence — rohanpaul_ai · 2026-10-31
- First machine-checked security proof for end-to-end x86 process execution published — banteg · 2026-10-31