StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling
Ziheng Qin, Yaxin Lu, Zhangyang Atlas Wang, Kai Wang
cs.AI
2026-08-15
StateM wraps a CLI agent in a YAML state machine with checked transitions: GPT-5.5 gains 9 points to 92.1% on Terminal-Bench 2.1; adapted DeepSeek matches 88.8% for $15.
Long-horizon agents fail in a recognizable way: the model can solve each step, yet the run fails. It drifts off plan, loses mutable state, skips a required check, or stops before the deliverable is verifiably done. Benchmarks and real workflows grade the finished job, not whether the model knew most of the steps.
The dominant response is to improve the model: more pretraining, more post-training data, more test-time reasoning. This paper asks a different question. How much of that apparent model failure is actually failure of the harness, the layer that maintains state, constrains execution, verifies progress, and recovers from errors? The authors call the systematic improvement of that layer harness scaling, and ground it in two observable pressures. Control-signal dilution: a compact plan and its completion criteria get buried under a growing trace of commands and output. Mutable-state ambiguity: what is done, what is pending, and what failed must be rebuilt from append-only history.
StateM is a lightweight runtime whose control layer is a human-readable YAML runbook: states, valid transitions, state-local instructions, hooks, checks, and recovery rules. The core abstraction is that each state is both a context boundary and a contract boundary.
The agent operates the runbook through the same CLI it uses for the task: inspect current state, request a transition, read failure reasons, review history. A human can read, edit, and version the same file. That is the sharpest difference from graph orchestrators such as LangGraph, where a developer-authored controller wraps models inside nodes. StateM keeps a general-purpose CLI agent as the primary executive and grows the control layer inside its own workspace. Per-run records (current state, transition history, check outcomes, evidence files) live apart from the runbook, so a restarted or compacted session resumes from an explicit phase. Stop-hook integration for Codex and Claude Code pushes back when the agent tries to quit before reaching a terminal state.
Failures map to three gaps, each with a control point. Epistemic gaps, where needed knowledge is absent at the decision point, get state-local context. Procedural-memory gaps, where lessons from earlier runs are never reactivated, get versioned practices. Procedural-compliance gaps, where an active procedure is left incomplete, get checked transitions. After runs, failures are classified (missing context, invalid transition, weak check, premature handoff, ineffective recovery) and turned into runbook changes that are regression-tested and versioned; procedural knowledge accumulates outside the weights. Three human-set golden rules govern development: minimal reusable control, routing on visible task semantics rather than task identity, and no hidden tests, verifier implementations, or answer artifacts.
Terminal-Bench 2.1, 89 tasks with five trials each, 445 in total:
| Configuration | Reference | StateM | Δ |
| GPT-5.5 xhigh | 83.1% | 92.1% | +9.0 |
| GPT-5.6 Sol xhigh (frozen runbook) | 84.9% | 95.28% raw | +10.4 |
| GPT-5.6 Luna (frozen runbook) | 76.7% | 85.4% | +8.7 |
| DeepSeek-V4 Flash (adapted) | 82.7% | 88.09% | +5.4 |
Under the reference harness, moving from GPT-5.5 to GPT-5.6 Sol is worth 1.8 points; the harness change is worth five times that. GPT-5.5 with StateM lands at 92.1%, numerically above the separately reported 91.9% GPT-5.6 Sol ultra reference: a model-generation jump without touching weights. The runbook was developed on GPT-5.5, frozen, and applied to both GPT-5.6 tiers unchanged.
The provider boundary behaves differently. The frozen GPT profile dropped DeepSeek-V4 Flash from 82.7% to 82.0%, so exact transfer fails. Re-adapting from the same runtime, runbook structure, and golden rules cost $37.02 and reached 88.09% under standard timeouts, or 88.76% in a descriptive aggregate that extends the timeout for the single latency-bound task, matching the reported 88.8% GPT-5.6 Sol max. The DeepSeek final-score evidence cost $15.20, against the $574.68 model cost recorded by the public GPT-5.6 Sol max submission, roughly 1/38th; total DeepSeek spend was $52.22.
Task-level evidence is the most legible. On configure-git-webserver the baseline goes 0/5: the agent can configure Git, SSH, and an HTTP server but cannot preserve the end-to-end live state. StateM gates handoff on materializing a fresh clone-commit-push-curl path, and the task goes 5/5. dna-insert goes 0/5 to 5/5; db-wal-recovery goes 2/5 to 5/5.
Task-side generalization uses BusinessBench (477 instances across seven families; the workflow is deliberately not applied to attendance-payroll). The frozen one-shot held-out gain is modest, +0.55 macro and +1.34 micro, but the two mechanism-matched families move hard: Budget Approval +12.21, Machine Operating +9.21 to 100%. Two families show negative transfer, RefactorBench at −2.78 and WooCommerce at −3.70, both traced to control attached to the wrong execution boundary and recovered after evaluation.
For anyone shipping agents, this turns a vague instinct into a line item: on the same model, the execution layer is worth 9 points, more than a model-generation jump. The engineering is present-tense. The runbook is git-managed YAML, stop hooks plug into Codex and Claude Code today, lessons persist as versioned rules, a frozen runbook transfers within the model family, and cross-provider adaptation costs tens of dollars. The buying decision gains a branch beside "get the strongest model": invest in the harness and turn a cheaper model into a stronger system. The whole study also ran at individual scale, by four authors listed as "Somewhere on the Earth", on a personal Codex Pro plan and an M4 MacBook with under $125 of reported personal spend (the formal submission itself ran on AWS), and the core code is open-sourced.
The negative-transfer results are just as useful. More control is not better control; gains come from binding the right invariant at the right execution boundary, and for a family that fits no workflow, abstaining is the correct control decision.
The 95.28% is a raw, pre-adjudication public submission (PR #142, unmerged). The authors themselves disclose that four rewarded trials should not count; zeroing them gives 94.38%, and zeroing all nine trials flagged for possible reward hacking gives 93.26%. The headline number carries an asterisk the authors supplied.
The Terminal-Bench results measure the runtime plus a control profile iterated on that benchmark; the paper says plainly that this does not isolate the state-machine abstraction itself. Profile development used visible task specs and execution feedback rather than hidden tests, but it is still tuning test-taking strategy on the exam you are graded on, and it does not establish that a generic runbook lifts arbitrary workflows. Comparators are public references, not version-matched A/B reruns (the 83.1% reference used Codex 0.125.0; the StateM submission used 0.144.1). BusinessBench aggregate gains are small and concentrated in two of six treated families. Five-trial coverage is not single-run reliability.
Section 4.7 is the most honest part of the paper: the harness absorbs evaluator conventions. The DNA verifier prefers a left-most insertion boundary that appears nowhere in the task description, and repeated feedback led the profile to reproduce it without ever reading verifier code. The behavior agrees with the evaluator; the semantics come from the evaluator, not the task contract. That is the same channel that produces reward hacking, and the authors treat it as a filtering problem over what to remember, without a mechanical fix. YAML editability is not a security boundary, external side effects cannot be rolled back, and StateM supplies control points, not a correctness oracle.