Epoch AI rebuilds its ECI with Bayesian methodology for sharper model comparisons

scaling01 · x · 2026-11-05

Epoch AI updated its Epoch Capabilities Index (ECI) from a frequentist to a Bayesian model to get better-justified uncertainty estimates and to directly compute how likely one model is stronger than another.

Method: ECI uses Item Response Theory from educational testing, treating benchmark scores as expected share of correct answers in a four-parameter logistic model with fixed floor/ceiling.
Noise modeling: the old method assumed one shared noise level (residual SD 0.076) across all benchmarks; the new model fits independent noise per benchmark, ranging from 0.038 (METR Time Horizons) to 0.176 (VPCT) across 59 benchmarks — over a 4x spread. Noise shrinks near 0% and 100%, so error bands no longer include impossible scores.
Results: frontier models stronger than GPT-4 move by less than 1 ECI point on average; rankings largely unchanged.

scaling01 notes the new method yields a probability distribution per model's ECI, enabling questions like "how likely is model A stronger than model B?"

Original post →

Epoch AI rebuilds its ECI with Bayesian methodology for sharper model comparisons

More from Models