ApprenticeBench: closed models hit 77% vs 22% for open, widening capability wall
ysu_nlp · x · 2026-11-19
ApprenticeBench — which tests whether an agent can learn to do an entire knowledge job end-to-end through continual learning — updated its leaderboard, exposing a huge closed-vs-open gap:
- Best closed models now beat human testers on both score and cost: Gemini 4 Argon scores 66% at $6.03/task vs the human tester's 51% at $7.21
- Frontier models from Anthropic, OpenAI, Google, and Meta all exceed the human baseline — all closed
- Closed vs open stands at 77% vs 22%, the largest gap yet on economically valuable knowledge work; the best open-weight models trail by 55 points and none show meaningful on-the-job learning
ysunlp comments: for agents to take on independent knowledge work, a static harness isn't enough — every company and job has its own workflows, conventions, software systems, and exception handling, so agents must learn on the job to specialize. That gap is widening, and open models face a capability wall they need to break through.
More from AGI Musings
- Researcher: Real scientific progress isn't rewarded, expect decades of incrementalism — gerardsans · 2026-11-19
- Forethought podcast: could an intelligence explosion run inside a data centre? — willmacaskill · 2026-11-19
- AI job displacement is really an asset/liability duration mismatch, argues thread — curious_vii · 2026-11-19
- 'Have all the famous humans been born?' Sparks debate on human agency in the AI era — zetalyrae · 2026-11-19
- Nathan Benaich proposes an 'AI Genius Bar' to help small businesses adopt AI — nathanbenaich · 2026-11-19
- Kevin Roose Interviewed 150 Insiders for New Book, Reviving Dario's 2017 'Blob of Compute' Memo — every · 2026-11-19