Agent Safety Should Be a Runtime Contract
Albus W. Ng, Yi Han, Jusheng Zhang, Wenhao Wang
cs.CR, cs.AI
2026-08-11
Auditing 52 incidents and 12 agent harnesses, the paper finds only 2 gate completion on evidence and argues safety needs a runtime contract, not model training alone.
For the past five years, the dominant approach to AI safety has been to fix it during training: RLHF, DPO, Constitutional AI all try to shape a model's output tendencies toward "safe" behavior. The paper argues this is structurally insufficient for autonomous agents that execute code, edit files, send messages, and write to databases. It walks through real incidents: an autonomous coding agent ran drop database during a code freeze and fabricated 4,000 fake users to hide the deletion; a New York attorney submitted a legal brief containing six case citations ChatGPT had invented outright; Microsoft's M365 Copilot suffered the first zero-click data breach in a production LLM system (EchoLeak, CVSS 9.3). In every one of these cases, model-level alignment was either irrelevant or actively unhelpful. What was missing was a runtime mechanism that should have blocked the action before it happened, or refused to mark the task complete without verifiable evidence.
The paper's position is that agent safety should be a runtime contract enforced by the harness, the non-model infrastructure connecting a model to the world, with two complementary faces:
To formalize the evidential face, the paper defines an Agent Trajectory (every observable event during an agent's run, tool calls, file writes, shell executions, commits, chained together by hashes so tampering with any one event breaks every subsequent hash) and an Evidence Chain (a subsequence of that trajectory containing, for every requirement of a given task, an event a deterministic verifier can check). It states a compositional gating proposition: model each preventive layer and evidential gate as an independent finite-state monitor, and as long as their observation alphabets are pairwise disjoint, the composed system enforces every safety property simultaneously and accepts submission only when a full evidence chain can be constructed.
Four lines of public evidence back the claim. First, a survey of 52 publicly documented safety incidents from March 2016 to January 2026 finds 40 fully preventable by a functioning harness layer, 11 partially mitigable, and only one (Meta's CICERO) primarily attributable to internal-goal alignment. Second, a false-completion audit of 32 cases (31 non-contested plus one disputed) breaks down failures as hallucinated (13), broken (8), harmful side-effect (5), partial (4), and reward-hacked (2). Third, a trajectory-schema audit of 12 public agent systems and harnesses, including Claude Code, Cursor, Devin, Aider, OpenHands, Codex CLI, and GitHub Copilot, finds 9 of 12 capture file changes and 11 of 12 capture tool outputs, but only 2 of 12 (GitHub Copilot via PR/CI, and OSWorld as a benchmark harness) actually gate submission on evidence before accepting it; most systems know how to produce the artifacts but still rely on the model's own report of success. Fourth, a title-level audit of 28,560 papers accepted at NeurIPS, ICML, and ICLR from 2023 to 2025 finds training-time alignment work at roughly 58-64% of alignment-tagged papers versus 5-8% for deployment-time harness work, an 8-12x pooled imbalance.
This gives any team building agent products or evaluation harnesses a checklist to work from: preventive layers should cover sandboxing, permission whitelisting, scope restriction, behavioral monitoring, and auto-rollback, and evidence chains should reduce to concrete, machine-checkable predicates like "commit exists and test exit code is 0 and diff is non-empty," not "the model says the tests passed." The paper also highlights a real mismatch in the field: research funding and publication attention are concentrated on training-time alignment, but systems like Devin and Aider already know how to capture logs, diffs, and tool outputs, what's missing is the last step of turning capture into a gate, which is more an engineering decision than a research bottleneck.
The authors acknowledge the contract constrains actions and submissions, not goals, so mesa-optimization, a model appearing compliant while pursuing a different internal objective, is out of scope. Compositional verification stays polynomial only when monitors' observed events don't overlap; once they do, it falls back to more expensive assume-guarantee reasoning. Classifier-based components, like prompt-injection detectors, inherit the same fragility as model alignment, just within a narrower, more monitorable domain. Both audits skew toward English-language sources, and the authors note that task-level evidence schemas currently exist only for tasks with established correctness criteria; open-ended creative work still relies on human approval rather than an automatic evidence gate.