NeoCognition's ApprenticeBench: Testing AI Agents That Learn on the Job Like Apprentices
yuxiangw_cs · x · 2026-09-11
- NeoCognition (with @ysunlp) releases ApprenticeBench: it situates agents in realistic job environments like human apprentices, evaluating multiple dimensions of continual learning in a single run.
- Key idea: apprentice-style learning isn't purely online, offline, or RL — it also involves learning the environment. ML has analogies for individual elements but no holistic framework. This paradigm separates prior knowledge from genuinely new learning and, the author argues, should become the new norm for quantifying continual learning.
- Current state: agents are already quite good at continual learning. Humans remain far more efficient, but agents already have an edge in meticulous attention to detail.
- Broader takeaways: the gap between AI's coding/math prowess and mundane failures isn't about intelligence but the ability to learn new things quickly on the fly. The author also predicts a "Dark Forest" era where ideas get scooped instantly, driving demand for proprietary agents that learn on the job while keeping trade secrets in-house — NeoCognition's core business.
More from coding & agent
- huggingface_hub 2.3.0 cuts 1M-file listings to 20s, adds live Bucket following — lhoestq · 2026-10-27
- Nous Research ships Advanced Delegation plugin letting Hermes Agent route subagents to different models — NousResearch · 2026-10-27
- A dentist with a 3D printer discovered OpenAI Codex — and never looked back — gabrielchua · 2026-10-27
- Stanford's PaperFormalizer turns research papers into Lean-verified versions with author in the loop — StanfordUniversity · 2026-10-27
- Hugging Face TRL makes score centering + masked importance sampling default in async GRPO — _lewtun · 2026-10-27
- a16z: agents follow tech's oldest law — chaos first, then rein it in — a16z Newsletter · 2026-10-27