AI eval leaders from OpenAI to NIST meet as benchmark ecosystem stays siloed
typewriters · x · 2026-10-28
The author co-hosted an SF dinner with SnorkelAI, gathering eval leads from OpenAI, Meta, Google DeepMind, NIST, METR, Epoch AI and dozens more. Key observation: despite years of rhetoric that AI standards would spark a race to the top, the private-sector and nonprofit teams actually building and maintaining benchmarks barely talk or collaborate on shared frameworks—without shared measurement knowledge, we can't know if AI systems meet standards. The dinner followed SnorkelAI's $30M open benchmarking grant program and preceded the first ARC Prize Research Summit; attendees on the nonprofit side hadn't even heard of Snorkel, which just raised a $350M Series E with a business built on shipping quality benchmarks.
Related event: AI Evaluators Gather for First Time, But Audit Ecosystem Remains Fragmented(2 posts)→
More from Safety
- Negative result: ignorance-based tamper-resistance only works for broad knowledge domains — StephenLCasper · 2026-10-28
- CATCH: ~200 examples of adversarial training make LLMs robustly honest across benchmarks — irinarish · 2026-10-28
- Anthropic disclosure: no insurer, or even the whole industry, could cover catastrophic AI risk — wfithian · 2026-10-28
- Tonawanda Seneca Nation opposes data center on wetland at People's AI Assembly — AINowInstitute · 2026-10-28
- Cloudflare launches PHP CTF bug bounty challenge to stress-test its WAF — dinasaur_404 · 2026-10-28
- agent-repo-preflight: Open-Source Read-Only Audit of AGENTS.md, MCP and Hook Configs Before Cloned Repos Hit Your Coding Agent — Digital-Fortress-Sec · 2026-10-28