Epoch tests six frontier models on its own real work: reliable on defined tasks, short of full automation
dl_weekly · x · 2026-10-21
Epoch AI's new "Epoch Automation Reports" gave six frontier models—including Claude Fable 5.1 and GPT-6 Astra—real tasks from Epoch's own work (graphics, research design) at max reasoning settings, then graded outputs against employee standards.
Key findings
- Fable 5.1 and GPT-6 Astra lead but fall short of automating Epoch-quality work; open-weight models lag further, struggling even on well-defined tasks.
- Models miss implicit conventions (visual style, audience tastes) even with ample reference material, or over-stuff outputs with information.
- Models spot promising research directions but lack the judgment to follow through on experiment design.
Why it matters: existing benchmarks focus on easily verifiable tasks—a subset of real work—and can overstate progress when models are optimized for them. This hybrid approach exposes capability gaps that conventional benchmarks miss, informing AI adoption decisions.
More from Models
- Kronos: open-source K-line foundation model trained on 12B candles, with a free AI trading setup guide — Roger_M_Taylor · 2026-10-21
- Asked Fable 5.1 why its game tooling is mediocre: it says NO, Fable 5 says 'hahaha maybe' — repligate · 2026-10-21
- Humor Elo leaderboard: Gemini 4.1 Arvon tops at 1573, GPT-5.1 Turbo last with 179 — jon_barron · 2026-10-21
- From 9 to 48 tokens/s: STRATA + CachyOS supercharges local Qwen Flash on an 8GB GPU — Ne00n · 2026-10-21
- David Ondrej interviews Ahmad Osman on running AI fully locally, DeepSeek 4.1 Flash and DGX Spark vs Mac Studio — David Ondrej · 2026-10-21
- Beam 1.0 ranks #1 among US open-weight models on Excel financial modeling benchmark EMB — bigblueboo · 2026-10-21