Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents
Zining Huang, Haoran Que, Hong Zeng, Ge Zhang, Zuo Wang, Jin Chen, Haodong Wang, Zhongfei Hou, Changxin Pu, Shen Yan, Wenhao Huang
cs.AI
2026-08-12
ByteDance Seed's Harness-IF scores coding agents rule by rule; all 12 frontier models drop 3.6-7.4 points on rules that oppose their unprompted defaults.
A deployed coding agent reads instructions from a stack: the harness's own defaults, a system prompt, tool descriptions, skill descriptions, project files such as CLAUDE.md, and the user's current request. Two families of benchmarks fail to measure whether it obeys that stack. Instruction-following benchmarks (the IFEval lineage) pack every constraint into the user turn, while coding-agent benchmarks (the SWE-bench lineage) score final task success. Operational rules such as branch naming, commit language, and output format fall between them.
A subtler flaw sits underneath. When a model obeys a rule, it may simply have been going to do that anyway. If the rule says write commit messages in English and the model defaults to English, crediting that as compliance inflates the score. Existing benchmarks cannot separate compliance from coincidence.
Harness-IF turns instruction following into a rule-level measurement problem, with three load-bearing design choices.
Accuracy spans 72.1-85.9% across the 12 builds; AP-Acc spans 66.1-78.6%, and every model scores lower on the against-prior subset.
| Build | Acc | AP-Acc | Gap |
| Claude-Opus-4.7 | 85.9 | 78.6 | +7.3 |
| GPT-5.5 | 83.1 | 77.0 | +6.1 |
| Claude-Sonnet-4.6 | 82.5 | 78.5 | +4.0 |
| Claude-Haiku-4.5 | 79.0 | 71.9 | +7.2 |
| GLM-5.1 | 78.8 | 75.2 | +3.6 |
| Seed-2.0-Pro | 73.6 | 66.1 | +7.4 |
The gap is not a constant that cancels in comparisons: it ranges from 3.6 to 7.4 points across the cohort, a twofold spread. Claude-Opus-4.7 leads all four columns and keeps the top spot, but the prior control exchanges three adjacent rank pairs.
The failure decomposition adds two practical findings. Of 8,440 failures, 77.1% are shortfalls (the agent failed to do what a rule demanded) against 20.8% oversteps, and the two classes fail at similar rates (23.8% vs 20.8%); the imbalance is exposure, not propensity. Verifiers tuned to catch excess behavior therefore address only about a fifth of the failure mass, since most failure is omission. By family, output control is the hardest (70.9%, the lowest-scoring family for 11 of 12 builds) and carries the largest failure share at 27.6%; by modality, commanding rules are hardest (76.0%) and preferences easiest (90.6%).
Two takeaways for anyone deploying coding agents. First, do not trust aggregate instruction-following scores: when a rule runs against the model's grain, actual compliance sits 4-7 points lower, so compare builds on prior-stratified metrics. Second, placement matters. In the counterbalanced E0 conflict pilot, system prompts, project files, and user instructions tie exactly for highest precedence, with tool descriptions (mean rank 3.78) and skill descriptions (4.56) last. The user instruction only tying for first contradicts the intuition that later placement in the prompt wins. Rules that must be followed belong in the system prompt or a project file, not buried in a tool description.
The paper is candid here. 86.8% of verdict rows involve an LLM judge whose labels shift under a judge swap (62.1% agreement, kappa 0.163), which the authors name as the dominant uncertainty; they therefore claim cross-model patterns, not adjacent-rank orderings, and item-clustered intervals leave every adjacent comparison unresolved. The 60 items were selected from an 80-item pool for discriminativeness, an admitted selection optimism. E0 ran on nine older builds; its pooled ordering survives a crossed bootstrap (9,652 of 10,000 resamples) but only 6 of 9 individual builds reproduce it exactly, so it is a cross-build tendency, not a universal hierarchy. AP-Acc is a behavioral stratification, not a claim about training provenance.