Dev urges OpenAI to make independent evals a real market, not a small panel of labs
arthurcolle · x · 2026-09-14
Developer arthurcolle replied to Sam Altman arguing that what most needs evaluation now is not the model in isolation but the harness and agent ecosystem around it across distributed AGI use cases — a surface a handful of orgs cannot cover, but a market can. On OpenAI's Independent Evaluators program, he noted it currently means METR, Redwood and a few others: "a panel, not an ecosystem." He wants independent evaluation to become a real market on top of the training pipeline, where orgs build, buy and sell evals for specific industry verticals, backed by a robust open-source evals community.
More from AGI Musings
- Security researcher: vibe coding will force software to consolidate around mature products — jd_pressman · 2026-10-31
- AI pushes circuit lower bound from 3.1n to 3.5n as UCLA-backed challenge aims at P vs NP — soumitrashukla9 · 2026-10-31
- What is work for? Cross-national survey of 8,234 adults on meaning in the AI age — soumitrashukla9 · 2026-10-31
- Which universities win top students may hinge on healthy AI-use boundaries, argues researcher — brianryhuang · 2026-10-31
- Researchers debate whether more inference compute can substitute for training compute — teortaxesTex · 2026-10-31
- Five agents can agree because they made the same mistake, krishnan warns with satire — krishnan · 2026-10-31