NeoCognition's ApprenticeBench: Testing AI Agents That Learn on the Job Like Apprentices
yuxiangw_cs · x · 2026-09-11
- NeoCognition (with @ysunlp) releases ApprenticeBench: it situates agents in realistic job environments like human apprentices, evaluating multiple dimensions of continual learning in a single run.
- Key idea: apprentice-style learning isn't purely online, offline, or RL — it also involves learning the environment. ML has analogies for individual elements but no holistic framework. This paradigm separates prior knowledge from genuinely new learning and, the author argues, should become the new norm for quantifying continual learning.
- Current state: agents are already quite good at continual learning. Humans remain far more efficient, but agents already have an edge in meticulous attention to detail.
- Broader takeaways: the gap between AI's coding/math prowess and mundane failures isn't about intelligence but the ability to learn new things quickly on the fly. The author also predicts a "Dark Forest" era where ideas get scooped instantly, driving demand for proprietary agents that learn on the job while keeping trade secrets in-house — NeoCognition's core business.
More from coding & agent
- Dev argues MCP is unnecessary: hand agents an API key and they'll get the job done — sull · 2026-09-11
- Dev to add AI-friendly collision checking CLI to OpenSCAD — _Stocko_ · 2026-09-11
- Cadenya: a year-long indie build of an MCP-powered agent runtime — littlebobbyt · 2026-09-11
- xAI team to livestream building a company from scratch with Grok Bot over three days — soleio · 2026-09-11
- ClawOS: an experimental open-source Arch-based OS with OpenClaw as its primary agent interface — heyneighbor · 2026-09-11
- Grok Bot adds Salesforce, HubSpot, Gong connectors plus installable sales-bot templates — XFreeze · 2026-09-11