HANDBOOK.md benchmark: top agents pass just 36.2% of 65 policy-heavy agentic tasks
bibryam · x · 2026-10-12
- Design: HANDBOOK.md simulates employees following company handbooks — 65 agentic tasks in self-contained company environments (mock email, chat, calendar, issue-tracking and commerce over MCP), each governed by an expert-written SOP of 20–124 pages across finance, medical billing, insurance, logistics and HR at 10 fictional companies.
- Memorization resistance: every task mutates one of 10 base handbooks, changing the rules grading depends on; scoring is fully deterministic with 824 programmatic rubric criteria.
- Findings: 30 configurations tested; the best scored 36.2% under strict grading, but allowing one failure nearly doubled top scores — agents suffer "near misses," so critical constraints need pass-or-fail evaluation.
- Model results (chart): Opus 4.8 (max) leads with 23% strict pass@1 and 50% pass@(N-1); GPT-5.5, GLM 5.2 and Gemini 3.5 Flash follow; Grok 4.3 and Nemotron 3 Ultra trail.
More from coding & agent
- lucasmeijer: code mode is a good solution when agents need new behaviors — lucasmeijer · 2026-10-12
- pi author: agent self-modification is really user-initiated, agents don't need to rewrite their harness — lucasmeijer · 2026-10-12
- Agent platform pivots to OpenAI-compatible hosting after users rejected its all-in-one SDK — Pitiful-Surround-285 · 2026-10-12
- OpenAI targets fully automated AI researcher by March 2028, doc reveals — imjustnewatai · 2026-10-12
- Linus Torvalds used LLM tools to decipher kernel code, found 3 more latent bugs — PointmanW · 2026-10-12
- Same Model, Different Harness: OpenCode Refuses Far More Than Codex in Side-by-Side Tests — Current_Balance6692 · 2026-10-12