Objectively measuring AI progress is hard — the problem may be the benchmarks themselves

maksym_andr · x · 2026-11-17

Concluding his thread, maksymandr argues that objectively measuring AI progress is hard. He is skeptical ECI/AECI captures the community's vibes: Opus 4.5 was a clear breakthrough that drove holiday vibe-coding, yet the AECI plot shows it underperforming the general trend. The issue isn't the ECI methodology but the underlying benchmarks — perhaps we just need better ones.

Original post →

More from Research

Research channel →