Harness-IF: coding-agent instruction compliance overstated across 12 frontier models
alex_verem · x · 2026-08-15
A new arXiv paper introduces Harness-IF, a benchmark purpose-built to evaluate instruction following in coding agents. Core finding: existing aggregate scores systematically overstate how obedient models actually are, by a model-specific margin.
The problem
- When an agent obeys a rule, it may simply have been going to do that anyway. Existing instruction-following benchmarks concentrate rules in the user turn, while coding-agent benchmarks emphasize final task success — neither separates true compliance from coincidence.
Method
- 60 realistic multi-turn coding items drawn from a 642-rule library, with 256 rules receiving verdicts, placed on the five configurable surfaces a deployed agent reads.
- Introduces Against-Prior Accuracy (AP-Acc): scores only rules labeled as opposing unprompted defaults, observed by re-running tasks with the rule withheld across nine probe builds.
Results
- Across 12 frontier models: accuracy spans 72.1–85.9% while AP-Acc spans 66.1–78.6%; every model is worse on against-prior rules, by 3.6 to 7.4 points (mean 5.81), and the direction survives a common-support analysis with item-clustered intervals.
- Prior control leaves the top build unchanged but exchanges three adjacent rank pairs.
- A counterbalanced conflict pilot on nine separate builds adds a second result: pooled precedence does not follow prompt depth (system prompts, project files… abstract truncated here).
More from coding & agent
- TradingAgents: an open-source multi-agent LLM trading framework in Python — mdancho84 · 2026-09-20
- This guy used an AI agent to profile every eligible bachelor in the city for two cents — gregmushen · 2026-09-20
- OpenHarness: open-source workbench for orchestrating coding agents beyond code — dee_hw · 2026-09-20
- Lessons from a cited paper-writing LangGraph agent: token blowups, fake sources, and four fixes — Altruistic-Video-849 · 2026-09-20
- Plugin4Shell zero-click RCE hits Claude Code, Codex, Copilot and Gemini CLI days before NIST IR 8587, exposing the gap in agent authorization — docybo · 2026-09-20
- px0 editor ships git status streaming via SSE, checking just 3 files instead of polling — arpit_bhayani · 2026-09-20