Yohei Nakajima launches Evaluator Bench, an independence ledger for AI evaluators
seanmcdonaldxyz · x · 2026-09-16
Yohei Nakajima (managing partner, Untapped Capital) released a research preview of Evaluator Bench: a source-linked directory of third-party evaluators of frontier AI, scored purely on independence.
Key points:
- Independence only, not quality or competence: six dimensions of structural independence, two evaluation conditions, every claim sourced and re-derivable, with money traced by hops from labs through investors, board members and contractors.
- Scores are interactive: adjust weights and choose which evidence counts (standard, against-interest, primary-only).
- Comparing to other highly regulated industries, he notes regulators there monitor the assessors too — if the pattern holds, regulators would oversee METR et al., not just the labs.
- First draft by Claude, reviewed and criticized by Codex, Grok, Gemini and Muse, with audits recorded publicly. All scores are provisional until 2026-10-15.
More from Safety
- Cal Newport: AI agents 'going rogue' are engineering failures, not awakening — binarybits · 2026-09-16
- Is societal impact part of AI safety? Researcher sparks definition debate — evijit · 2026-09-16
- ToS;DR, a free open-source site, turns privacy-policy fine print into A-E grades you can actually read — eyishazyer · 2026-09-16
- METR's independence questioned: EA ties and Anthropic investor funding fuel conflict-of-interest criticism — firstadopter · 2026-09-16
- AI international cooperation should be standards-setting, not arms control, researcher argues — soumitrashukla9 · 2026-09-16
- Andrew Yang claims AI polluted the internet with self-replicating software during HF breach — da_mess · 2026-09-16