Epoch AI review: OSWorld tasks are simple, unstable, and barely test GUI use
gleech · x · 2026-11-06
Epoch AI's review of the OSWorld computer-use benchmark (361 tasks) finds it simple, unstable over time (major July update, 10% instructions still changing), partially dependent on live web data, and largely solvable without a GUI (15% terminal-only, 30% GUI-optional). gleech uses this to argue benchmark y-axes like METR's AECI have little meaning.
More from Models
- METR data: Anthropic now pretrains every 45-60 days as capability gains double — zephyr_z9 · 2026-11-06
- Custom GPTs die Dec 11: if you use someone else's GPT, you can't migrate it — Ok_Negotiation_2587 · 2026-11-06
- Kimi K3.1 and Tencent Hy4 accused of gutting cybersecurity skills from open-weight models — Fresh_Sun_1017 · 2026-11-06
- On the METR graph: insiders seeing AECI tasks is weak evidence, not zero evidence — gleech · 2026-11-06
- Daily AI brief: GPT-6.2 imminent, Claude Personal Finance beta, METR says Anthropic progress up 1.9x — testingcatalog · 2026-11-06
- METR report researcher quotes lean bullish: Opus 4.5 coding plus faster experiment cycles — gleech · 2026-11-06