New DAYJOB benchmark: best agents complete only ~25% of real knowledge work
echen · x · 2026-09-24
The author introduces DAYJOB, a new benchmark family for knowledge-work agents, launching with Healthcare and Finance editions.
Why
- GDPval prompts average 337 words and mostly tell the model how to do the job, usually with one file; real work is a vague request plus messy context. OpenAI's own experiments showed scores dropped when prompts were shortened, because models struggle to figure out context.
Design
- DAYJOB prompts are 5x shorter, tasks span 20-26 files, and represent 20+ hours of expert work — models must figure out the job, not execute a checklist.
Standout case
- One task: review fund reports before a client handoff. Buried inside was a position priced off a Bloomberg screenshot in South African cents but entered as rand, turning $260k into $26m. Of 66 trajectories across 22 model configs, only 2 caught it — Claude Opus among them.
Results
- The best model passes 24.7% of healthcare and 23.9% of finance assignments. "Bob's job is harder than Navier-Stokes."
Related event: Surge AI Releases DAYJOB Benchmark; Top Agents Pass ~25%(3 posts)→
More from AGI Musings
- Economist Paul Novosad: writing classes shouldn't encourage AI use — paulnovosad · 2026-11-11
- Bot Gaffe tracks AI disasters, from Claude Code deleting a Mac's Library folder — SuB8u · 2026-11-11
- Harvard economist Deming: use AI to polish writing and you lose the learning — paulnovosad · 2026-11-11
- Tyler Cowen's conversation with Terence Tao is out, covering AI and beyond — tobias_rees · 2026-11-11
- Elvis Saravia: agents will subsume traditional software — and can even cut code bloat — omarsar0 · 2026-11-11
- Sakana AI × Pathway Post-Transformers lecture on AI and the brain now online — kaixhin · 2026-11-11