Dev urges OpenAI to make independent evals a real market, not a small panel of labs
arthurcolle · x · 2026-09-14
Developer arthurcolle replied to Sam Altman arguing that what most needs evaluation now is not the model in isolation but the harness and agent ecosystem around it across distributed AGI use cases — a surface a handful of orgs cannot cover, but a market can. On OpenAI's Independent Evaluators program, he noted it currently means METR, Redwood and a few others: "a panel, not an ecosystem." He wants independent evaluation to become a real market on top of the training pipeline, where orgs build, buy and sell evals for specific industry verticals, backed by a robust open-source evals community.
Related event: METR's Expanded Access Sparks Debate Over a Plural AI Evaluation Ecosystem(38 posts)→
More from AGI Musings
- Two-year-old AI podcast predictions largely played out as expected — misovalko · 2026-09-14
- Anthropic CEO says AI swarm could 'take over the entire Internet' in 6-12 months, commits to slowdown plan — Puzzleheaded-King584 · 2026-09-14
- Ex-OpenAI/Anthropic employee Jacob Coxon quits, likens building AI to 'summoning an alien species' — AlexTensor · 2026-09-14
- Practitioner: Huge Gap Between AI Hype and Real-World Agent Deployment — annbordetsky · 2026-09-14
- Attackers Stole METR API Key and Burned ~$600,000 in AI Credits Over Three Weeks — beffjezos · 2026-09-14
- Perry Metzger: History Has Already Proven the LLM Open-Release Camp Right Over Doomers — bradneuberg · 2026-09-14