Harness Arena: open-source blind benchmark for AI coding agent harnesses
Due_Armadillo_8744 · reddit · 2026-09-02
A Reddit developer open-sourced Harness Arena (MIT license) to fix agent harness comparisons that change too many variables at once — different model, prompt, tools, environment and task.
- Design: harnesses get the same task, run independently in isolated workspaces, deliverables are collected, outputs shown anonymously, scored before identities are revealed, and results feed an Elo leaderboard
- Integrating/testing Claude Code, Codex CLI, Hermes, OpenClaw, OpenCode and OnDemand, with free Kimi, Qwen and GLM options for experiments
- Open methodology questions: same model? same reasoning effort? equal token/context budget, identical tools/MCP servers, wall-clock/retry/cost limits? Should latency and cost be scored or shown separately? Category-specific leaderboards vs one overall board?
- Repo: github.com/Ondemand-OSS/harness-arena; the author is soliciting input on what methodology would make a harness-vs-harness benchmark trustworthy
Related event: Harness Arena: Open-Source Blind Benchmark for Coding Agents(2 posts)→
More from coding & agent
- LLM one-shots a Minecraft 3x3 piston door by building its own Linux test infra — Vjeux · 2026-10-18
- Dev builds Mumbai-inspired 3D manga background generator from OpenStreetMap data — CurieuxExplorer · 2026-10-18
- Solo founder builds a money-printing data service on Claude's Pi Durable harness — MicahBerkley · 2026-10-18
- Dev builds Arch-based OS with custom desktop and browser using Claude Code — Snoo-10493 · 2026-10-18
- Pi Durable went through 7 rewrites yet agent session files still lack a standard — ricklamers · 2026-10-18
- Five SaaS agent workflows fail the same way — and it's never model intelligence — usehive · 2026-10-18