Investor: Model companies pay benchmark firms to test multiple versions pre-launch
deedydas · x · 2026-10-01
Investor deedy reveals that most if not all model companies pay benchmark firms to test their models — and multiple versions of them — ahead of launch, which is how benchmarking companies make money.
On pricing, he argues the exact number matters less than landing within roughly ±25% of the frontier range. The metric is gameable via arbitrarily high pricing, and companies can't skip benchmarking without counter-signaling. But if a company prices at the frontier and sees meaningful usage uptick, he buys that it's a good model.
Related event: Investor Deedy Says AI Benchmarks Are Gamed—Trust Pricing Instead(5 posts)→
More from Venture
- Vinod Khosla offered founder co-founders' stock if he fired them; founder blocked his number — maddiehfaulkner · 2026-10-01
- How companies can earn $1M+ licensing workflow data to AI via micro1 — rgc4444 · 2026-10-01
- Lago joins NVIDIA Inception to help AI companies monetize products and sell to agents — mattturck · 2026-10-01
- Meta Claimed $3.9B Research Tax Credit by Labeling AI Data Centers as 'Experiments' — mkheck · 2026-10-01
- Cerebras silence over SemiAnalysis claims questioned as insiders dump $CBRS stock — firstadopter · 2026-10-01
- growth.engineer launches as an open-source catalog of GTM workflows you can copy into any agent — TAbrodi · 2026-10-01