ApprenticeBench claims strongest model separation: 72% vs 18% job completion for near-tied models
ysu_nlp · x · 2026-09-11
Quoting the ApprenticeBench launch thread, the author argues it may be the most discriminating agent benchmark yet, showing a step change for Fable 5.1 and GPT-6 Astra.
- The benchmark tests computer use plus continual learning on a real job: agents must learn company conventions from past bills, adapt to policy changes, and operate the company's software
- Key contrast: Fable 5.1 and Opus 5 sit just a few points apart on AA, Terminal-Bench 4.0 and CursorBench 4.0, yet score 72% vs 36% here
- The gap is even starker for open-weight models: Fable 5.1 (53) and Kimi K3 (44) on AA score 72% and 18% respectively
- Rationale: individually small model gaps compound when all challenges stack, turning into a massive difference in actually getting the job done
More from AGI Musings
- Almost everyone underestimated AI progress: forecasts off by years on benchmarks and revenue — ben_j_todd · 2026-10-27
- Goldman finds statistically significant negative link between AI adoption and US employment growth — ben_j_todd · 2026-10-27
- a16z: World-changing tech makes a chaotic entrance — AI agents are following the same path — a16z · 2026-10-27
- NEJM AI on medical education: mastery requires performing without the machine — BraydonDymm · 2026-10-27
- Dev warns abliterated models could wreck software supply chains with 1000+ transitive dependencies — nitarshan · 2026-10-27
- Delete your priors: why every SOTA model drop should reset how you think the world works — zack_overflow · 2026-10-27