Benchmark builders from private sector and nonprofits aren't talking to each other
typewriters · x · 2026-10-28
The poster co-hosted an SF dinner with Snorkel AI for people building and leveraging evals, bringing together practitioners from the private sector and nonprofits who develop, deploy and maintain benchmarks. The striking observation: after years of claims that creating AI standards would drive a race to the top, the experts actually building benchmarks were not talking to one another or collaborating on scalable frameworks.
Related event: AI Evaluators Gather for First Time, But Audit Ecosystem Remains Fragmented(2 posts)→
More from Companies & People
- Nathan Lambert: an effective org is worth multiples in compute for training frontier models — natolambert · 2026-10-28
- Nous Research sees nonstop signups after integrating Memelord's MCP — NousResearch · 2026-10-28
- Anthropic disclosure: no insurer, or even the whole industry, could cover catastrophic AI risk — wfithian · 2026-10-28
- One teacher becomes her whole network's 'FDE' after learning Claude for teaching tools — BenBajarin · 2026-10-28
- Dario: Those who called his GPT-2 fears 'pseudoscience' in 2019 now all apply to AI labs — irinarish · 2026-10-28
- After mom's missed cancer diagnosis, founder builds AI agents for patient follow-up — aakashgupta · 2026-10-28