Same model, four coding harnesses: harness choice moved scores more than expected
Individual-Quote-958 · reddit · 2026-10-24
A Reddit user (Polarline's founder) ran a controlled experiment holding GPT-5.5 fixed while comparing Claude Code, Codex, OpenCode, and his own Scout (an OpenCode fork) across 5 tasks, 3 trials each.
Key findings:
- The harness mattered more than expected, especially on multi-step work
- His own harness lost on numeric discipline, and every harness struggled with inventing numbers
He acknowledges the limitation: it's his own small benchmark with self-picked tasks, and he's asking what a harness benchmark should test beyond this.
More from coding & agent
- ClimateTriage MCP Server lets AI agents search open-source climate issues — modelcontextprotocol · 2026-10-24
- NoonAI DIS launches verified MCP for automatic face detection and de-identification — modelcontextprotocol · 2026-10-24
- Dev adds a 'Non-goals' section to agent work orders to stop unsolicited refactors — HaktanSuren · 2026-10-24
- How do you turn a real agent workflow into a useful evaluation set? — Super_Tough_4997 · 2026-10-24
- Glance ships MCP server so AI agents can create and update native iPhone Home Screen widgets — Dense-Map-406 · 2026-10-24
- Microsoft open-sources Agensh, a decentralized multi-agent harness scaling to 1,024 agents — donglixp · 2026-10-24