$1M into a public benchmark gets labs to spend $100M+ hillclimbing it
_jasonwei · x · 2026-11-20
Mercor CEO Brendan Foody observes that investing $1M in a public benchmark drives labs to spend $100M+ hillclimbing it, citing rapid model gains on benchmarks built with Harvey, Cognition, Sierra and Ramp over 6 months.
Jason Wei extends the point: when AI can optimize anything we quantify, much of coming value creation will come from inventing new ways to measure the world — benchmarks are becoming an industry of their own.
More from AGI Musings
- Garry Tan backs Bostrom's 'if nobody builds it, everyone dies', urges taking Deep Utopia seriously — garrytan · 2026-11-20
- Superforecasters put 33% odds on a week-long outage hitting 1M+ people in an OECD country by 2028 — NathanpmYoung · 2026-11-20
- Sentinel brief 'Consider Cyberarmageddon' extrapolates cyberattack growth to societal failure — NathanpmYoung · 2026-11-20
- Google researcher ibab: models trained on humanity's commons should be open-sourced — ibab · 2026-11-20
- Stanford team at NeurIPS 2026: analogical reasoning fixes LLM mode collapse in science — james_y_zou · 2026-11-20
- Founder claims 'LLM arbitrage': spotted 17 counterparty errors in one day — StewartalsopIII · 2026-11-20