Benchmarks get gamed: vendors sell solutions to labs, so 'emergence' is mostly training

sebkrier · x · 2026-11-08

Argues the "emergent properties" discourse is overblown: data vendors pick or build public benchmarks where frontier models do poorly, pay humans to solve them (or simulate key aspects), and sell the data to big labs — guaranteeing models eventually improve on any public benchmark. Rule of thumb: assume nothing is emergent without third-party-reproducible evidence. The quoted post claims GPT-6's robotics skills come from robotics-specific training data, not emergence.

Original post →

Benchmarks get gamed: vendors sell solutions to labs, so 'emergence' is mostly training

More from Models