Harness-IF Benchmark: AI Coding Agents Don't Truly Follow All Rules
omarsar0 · x · 2026-08-14
While current coding agents follow user-defined rules (like those in AGENTS.md), they often do so because they were going to act that way anyway. To separate genuine instruction-following from coincidental behavior, researchers introduced Harness-IF.
Evaluation Method:
It scores 256 rules one at a time from execution evidence, then re-runs every task with the rule withheld across nine probe builds to identify which rules actually oppose the model's defaults.
Key Findings:
- Accuracy Drop: Across 12 frontier models tested, raw accuracy runs between 72.1% and 85.9%. Once coincidences are stripped out (Against-Prior Accuracy), accuracy drops by 3.6 to 7.4 percentage points.
- Precedence Inversion: Precedence does not follow prompt depth. System prompts, project files, and user instructions all outrank tool and skill descriptions.
More from coding & agent
- Optimizing Agent Costs: Use a Lightweight Classifier Before Expensive Agents — hwchase17 · 2026-08-14
- 130k-Star Open-Source Repo: 100+ AI Agents and RAG Apps — Saboo_Shubham_ · 2026-08-14
- Building Entire SaaS in Under a Day: The Lovable Workflow That Works — PrajwalTomar_ · 2026-08-14
- Tool Use, Not LLM Inference, Will Become the Agent Latency Bottleneck — soumitrashukla9 · 2026-08-14
- Vercel Offers GLM 5.2 Model Free for eve Agents Until August 27 — cramforce · 2026-08-14
- X Open-Sources Recommendation Algorithm, Developer Uses Grok to Analyze Ranking — prasenx · 2026-08-14