Benchmark weighting fight: coding-only ECI shrinks open-closed model gap to ~2 months
wenhaocha1 · x · 2026-11-02
Quoting scaling01's claim that Chinese open models still trail Mythos Preview (161 ECI) after 7 months, @wenhaocha1 offers a technical critique of the ECI: unlike indices such as AA, GENERAL ECI doesn't manually weight benchmarks, making conclusions highly sensitive to weighting choices.
- His coding-only ECI chart shows closed models (Claude Opus 4.8, Opus 5.5) leading while open models (Kimi K3, DeepSeek-R1) close in; if coding matters most, the open-closed gap reads as only 2 months.
- He also sketches a leave-one-category-out refit approach to see how much the gap shrinks when each benchmark category is removed.
- Takeaway: "months behind" claims from aggregate indices depend entirely on how benchmarks are weighted.
More from Models
- Reflection to host Reddit AMA on Nov 3 about Beam, its first open-weight model — Aggressive-Sign-8167 · 2026-11-02
- Cursor Pro+ user burns through all quotas, says forced to upgrade to $200 Ultra plan — pramodk73 · 2026-11-02
- User regrets $500 OpenAI plan: 'ultra fast' is actually buggy computer use — imjustnewatai · 2026-11-02
- Users report Astra 6 quality collapse: endless apologizing loops, worse than models from a year ago — Icy-Way3920 · 2026-11-02
- GLM-5.3 repeatedly violated 'no memorized code' rule in security evals, Irregular finds — shoumikgoswami · 2026-11-02
- Tuning 0.001% of Gemma 3 270M's weights lifts sort accuracy from 1% to 97% — dejanseo · 2026-11-02