29 bugs in benchmark history have never been fixed by any model, says Pawel Huryn
PawelHuryn · x · 2026-10-13
In a benchmark-design discussion, Pawel Huryn notes that across all runs in benchmark history, 29 bugs have never been fixed by any model. He endorses filtering for the 105 tasks older models missed: easy bugs bunch every score together, and the hard tail is where a ranking actually separates models.
Related event: Pawel Huryn Benchmarks Muse Wonder 1.0: GPT-6 Astra Leads at 45 Points(5 posts)→
More from Models
- Gemini 4 Argon reportedly drops as user laments missing the moment — Angaisb_ · 2026-10-13
- Ask Claude for its preference instead of a recommendation and it turns non-moralistic — austinc3301 · 2026-10-13
- Muse Wonder scores 93.4 on Extended NYT Connections benchmark, among priciest tested — zero0_one1 · 2026-10-13
- ScaleDown launches Classify: 95% intent-detection accuracy at 1,810x lower cost — testingcatalog · 2026-10-13
- Next-gen laptops need a 35B MoE with ~9B active params, says dev — nothing above 3B exists — draginol · 2026-10-13
- Zuckerberg reveals Meta's frontier research team is only about 100 people — rohanpaul_ai · 2026-10-13