Deedy: model companies pay benchmark firms to pre-test models — price is the only honest signal
deedydas · x · 2026-10-01
Investor Deedy claims most if not all model companies pay benchmark companies to test their models — including multiple pre-launch versions — and that's how benchmarking firms make money.
He adds that in "late-stage AI model capitalism," frontier models must win benchmarks to launch, so those numbers mean nothing. The only signal to trust is price: if a model is priced high, it's good; if not, it's benchmaxxed.
Related event: Investor Deedy Says AI Benchmarks Are Gamed—Trust Pricing Instead(5 posts)→
More from AGI Musings
- 80,000 Hours' first debate: AI takeover vs human power grabs, with Katja Grace and Tom Davidson — KatjaGrace · 2026-10-01
- Tom Davidson updates after 80k debate: AI takeover may be the bigger risk than human dictators — KatjaGrace · 2026-10-01
- Seth Lazar: We're at the Threshold of 'Whack-a-Mole AGI'—and Must Decide Whether to Build It — sethlazar · 2026-10-01
- Linguist Argues LLMs Could Trigger the Next Theory of Evolution — begusgasper · 2026-10-01
- 1000-agent swarms, not single chats, could unlock novel AI-driven science — arjunrajlab · 2026-10-01
- Is closing a Claude tab murder? Consciousness thought experiment sparks debate — repligate · 2026-10-01