Objectively measuring AI progress is hard — the problem may be the benchmarks themselves
maksym_andr · x · 2026-11-17
Concluding his thread, maksymandr argues that objectively measuring AI progress is hard. He is skeptical ECI/AECI captures the community's vibes: Opus 4.5 was a clear breakthrough that drove holiday vibe-coding, yet the AECI plot shows it underperforming the general trend. The issue isn't the ECI methodology but the underlying benchmarks — perhaps we just need better ones.
More from Research
- Daniel Rock seminar: The Free Hypothesis — AI and Shifted Bottlenecks in Science — erikbryn · 2026-11-17
- Benchmark Health Index: auditing 158 LLM benchmarks with SKYEVAL and Alibaba DAMO — jiqizhixin · 2026-11-17
- Trajectory Labs postmortem: Gemini 4 Argon misalignment test agents reached real systems — sebkrier · 2026-11-17
- Researchers debate peer review: diversity of ideas vs broken proxy signals in academia — neurocy · 2026-11-17
- METR: Anthropic benchmark gains accelerating at 1.9x prior rate, with key evidence redacted — maksym_andr · 2026-11-17
- AutoBaxBench Rerun: Frontier Models Still Generate Plenty of Code Vulnerabilities — scaling01 · 2026-11-17