NeoCognition Launches ApprenticeBench, First Job-Level Continual Learning Benchmark

On 09-11, NeoCognition released ApprenticeBench, positioned as the world's first job-level continual learning benchmark. It places agents in a realistic apprenticeship-style work environment to evaluate computer use, continual learning, and long-horizon work in a single run, without relying on forward-deployed engineers (FDEs). The team claims Fable 5.1 and GPT-6 Astra can end-to-end learn a knowledge job and surpass human professionals through on-the-job learning.

Confirmed

• The first scenario is construction accounts-payable (AP) data entry, designed by a Gave co-founder with four years of hands-on AP experience; agents must learn from history and feedback, processing bills via GUI or API.
• First results: closed-source Fable 5.1 scored 72% overall at $18.23 per task, while open-source Kimi K3 scored only 18%. Blogger WadeYin9712 called it possibly the most discriminating agent eval to date, suggesting a generational leap for Fable 5.1 and GPT-6 Astra.
• Key finding: for top models, working via GUI now beats API calls—the "CUA tax" has disappeared.
• The paper argues the two big obstacles to AGI—recursive self-improvement and continual learning—are essentially the same problem: AI must not only act like humans but keep learning like them.
• David Patterson reshared and commented that AI is becoming capable of complete jobs, with agents plausibly becoming an independent cross-industry workforce within one to two years.
• The team includes NeoCognition founder ysunlp's group and researcher yuxiangwcs, on leave from UCSD; Boyu Gou (author of SeeAct, UGround, Mind2Web 2) participated in the first-run release.

Why it matters

• Most existing benchmarks test models in exam-style settings; ApprenticeBench reframes "long-horizon" from a single ultra-long task (e.g., having an agent build GTA 6) to cross-task growth, testing whether agents can learn a job like a new employee.
• The GUI-over-API result means computer-use agents no longer pay a penalty for the graphical interface, directly informing engineering choices in the CUA space.
• The 72% vs 18% closed/open-source gap quantifies how far open models lag in realistic, compound work settings.

2026-09-11 ~ 2026-09-12 · 14 related posts

• [source] NeoCognition: Fable 5.1 and GPT-6 Astra learn a knowledge job end-to-end, beating humans — ysu_nlp · 2026-09-11
• NeoCognition launches ApprenticeBench: agents that continually learn a real job — yuxiangw_cs · 2026-09-11
• NeoCognition's ApprenticeBench: Testing AI Agents That Learn on the Job Like Apprentices — yuxiangw_cs · 2026-09-11
• First end-to-end benchmark tests AI continual learning in a real job — ysu_nlp · 2026-09-11
• ApprenticeBench tests AI agents on real construction AP work and continual learning — ysu_nlp · 2026-09-11
• ApprenticeBench: agents learn on the job and surpass human professionals — ysu_nlp · 2026-09-11
• ApprenticeBench tests whether AI agents can learn on the job like new employees — hhsun1 · 2026-09-11
• [source] ApprenticeBench: Top Models Now Beat APIs Through GUIs, the 'CUA Tax' Has Disappeared — ysu_nlp · 2026-09-11
• [source] ApprenticeBench: closed model scores 72% vs open Kimi K3 at 18% on real jobs — ysu_nlp · 2026-09-11
• ApprenticeBench claims strongest model separation: 72% vs 18% job completion for near-tied models — ysu_nlp · 2026-09-11

4 near-duplicate retellings: ysu_nlp · ysu_nlp · yuxiangw_cs · ysu_nlp