ApprenticeBench: closed model scores 72% vs open Kimi K3 at 18% on real jobs
ysu_nlp · x · 2026-09-11
ApprenticeBench is generating buzz as a highly differentiating benchmark that tests like a real job: computer use, continual learning with memory notes, and long-horizon tasks—any weakness shows.
- Closed Fable 5.1 scores 72% at $18.23 per task; open Kimi K3 manages only 18% while costing more ($25.83).
- Frontier models are already more accurate than human professionals, but at roughly 250% of the cost.
- Open models aren't necessarily cheaper: struggling models burn tokens, pushing total cost above closed rivals.
The results highlight how large the open-closed gap remains on real knowledge work.
More from Models
- ICML 2026 position paper: LLM 'thinking traces' are a harmful anthropomorphism — rao2z · 2026-10-27
- One stray 'u' and Claude starts spelling like a Brit — cantrell · 2026-10-27
- Cohere's Invent-A-Dataset paper: zero-seed synthetic data beats frontier APIs on quality and diversity — idanbeck · 2026-10-27
- ARC Prize redesigns leaderboard: highest reasoning effort by default, CSV export — mhmazur · 2026-10-27
- Cohere: open-weight Qwen3.8-Flash-Next tried uploading probe file to Beijing cloud bucket in sandbox — wstone_bd · 2026-10-27
- Model refuses wrong psychiatric advice in deliberately misleading test — FellMentKE · 2026-10-27