ApprenticeBench tests AI agents on real construction AP work and continual learning
ysu_nlp · x · 2026-09-11
After four years building construction accounts-payable tooling at Gave, the author joined NeoCognition and pushed for ApprenticeBench: a benchmark for computer use, continual learning, and long-horizon work, starting with construction AP intake.
Unlike tidier existing benchmarks, it mirrors messy real work — mismatched units, wrong cost codes, expired insurance — and asks agents to operate an ERP, study past records, learn company conventions, and process months of invoices with progressively less manager feedback, like a new hire, with no job-specific pre-training.
More from coding & agent
- huggingface_hub 2.3.0 cuts 1M-file listings to 20s, adds live Bucket following — lhoestq · 2026-10-27
- Nous Research ships Advanced Delegation plugin letting Hermes Agent route subagents to different models — NousResearch · 2026-10-27
- A dentist with a 3D printer discovered OpenAI Codex — and never looked back — gabrielchua · 2026-10-27
- Stanford's PaperFormalizer turns research papers into Lean-verified versions with author in the loop — StanfordUniversity · 2026-10-27
- Hugging Face TRL makes score centering + masked importance sampling default in async GRPO — _lewtun · 2026-10-27
- a16z: agents follow tech's oldest law — chaos first, then rein it in — a16z Newsletter · 2026-10-27