Session-Bench v1 ranks 12 coding harnesses: same bug fix costs 19 KB vs 697 KB on disk

jazzy8alex · reddit · 2026-10-10

Session-Bench v1 runs one fixed task (failing check, correction, file edit, re-run) across 12 coding agent harnesses, 3 runs each, then scores 31 facts against an independent witness (live output stream, ledger, file hashes) — 100 points, all recomputable from published session files.

Rankings: DeepSeek Harness 96.9, Pi 96.4, Copilot CLI 96.0, OpenCode 94.4, Kimi Code 91.2, Claude Code 89.3, OpenClaw 88.1, Codex 87.5, Hermes 86.4, Claude Desktop 85.6, Antigravity 82.7, Cursor CLI 78.1.

Key findings:

Nobody loses the conversation: 11/12 score full marks on fidelity; the 19-point spread comes from elsewhere.
Same fix writes 19 KB (Pi) vs 697 KB (OpenClaw), where sessions are only 3% of the file — pasting raw sessions into a model costs 5,000 tokens vs 174,000 for the same work.
Four harnesses never store an event once: OpenCode duplicates message parts 1-5x in an event table, Kimi writes each prompt 3x, so raw-file message counts over-count.
Only 4 harnesses record token usage that adds up to a stated total; Cursor CLI stores no token count at all.
Antigravity and Cursor CLI store protobuf in SQLite without a published schema — sqlite3 opens the file but can't read the session.

Stated limits: one synthetic task, one machine, different models per harness (grades files, not models); some scoring rules written after first captures, 3 waived/corrected post-results (OpenCode first scored 97.4/1st, revised to 94.4/4th after outside review); reviews by AI agent sessions plus four rounds of external model review, no second human.

Original post →

Session-Bench v1 ranks 12 coding harnesses: same bug fix costs 19 KB vs 697 KB on disk

More from coding & agent