Doc2Harness benchmarks LLMs on harness engineering: GPT-6 Astra tops at 53.05%, still failing half
Yuyao Ge · hf · 2026-10-15
Researchers from ICT CAS, Tsinghua and Peking University introduce Doc2Harness, the first deterministic benchmark measuring long-horizon harness engineering—whether LLMs can build a complete coding-agent harness from an interface-only contract and a development document consolidating twelve frontier harnesses.
- Two-stage design: a build model constructs a candidate harness in an isolated container; the frozen harness then runs separately, driven end-to-end by a scripted mock model, scored across four observable channels (model requests, files on disk, exit codes, command output) with positive controls ruling out vacuous passes.
- Across 19 frontier models, 3 builds each on 328 test cases, GPT-6 Astra achieves the highest pass rate at 53.05% vs. 27.44% for the weakest—so even the best model fails nearly half the cases, with failures clustered around how cross-cutting functionality is split across modules.
More from coding & agent
- Player-1: RL turns VLM agents into GTA V players with zero game-specific training — yshan2u · 2026-10-15
- Agentic engineering vs AI engineering: one builds software with agents, the other builds agents as the product — Al_Grigor · 2026-10-15
- Cursor + Gemini 4 Argon builds a multiplayer 3D capture-the-flag game in one go — FinanceYF5 · 2026-10-15
- Claude Opus 5.5 levels up codegen but its docs and PR prose are 'unusable' — indragie · 2026-10-15
- Gary Marcus flags AI agent fatigue: managing coding agents burns out engineers — GaryMarcus · 2026-10-15
- The thrill of building isn't gone — you're just not stopping to enjoy it — _jaydeepkarale · 2026-10-15