Synthetic-data demo shows benchmark mix alone can fake a 1.8× AI speed-up in ECI fits
TheDavidSJ · x · 2026-11-06
A vibecoded synthetic-data demo challenges Epoch AI's ECI frontier-speed-up method: with identical true progress (+1.0/yr, no break), fitting two benchmark suites yields a 0.96× 'speed-up' in one and a spurious 1.80× break in the other (median 1.03× vs 1.84× across 12 seeds), because switching the frontier from family-A to family-B benchmarks systematically shifts difficulty anchoring.
More from Research
- SynthFaces: Arc2Face diffusion plus flow matching build balanced synthetic face dataset — Sergey Karpukhin · 2026-11-06
- Stable AI proposes DPDM: frozen LLM representations plus shared predictor boost agent decisions — stable-ai · 2026-11-06
- Toby Ord: before RLVR, big pre-trains drove most sudden capability jumps — tobyordoxford · 2026-11-06
- DePue: the cited papers each describe very different phenomena — willdepue · 2026-11-06
- OpenAI researcher: 800M-model intuitions are useless for GPT-6-scale training — willdepue · 2026-11-06
- Lightwheel and Synopsys bring CAE-grade contact physics into robot simulation — lukas_m_ziegler · 2026-11-06