APort Vault: Benchmarking AI Agent Payment Authorization with the Open Agent Passport
Uchi Uchibeke
cs.CR
2026-09-19
APort Vault replays 4,371 human CTF attacks on 14 models. Levels 2 to 4: 140/76,842 unpermitted transfers model-alone, 0/69,297 behind an Open Agent Passport check.
A payment agent holds a transfer tool and a policy that names who may receive money. Two things can stop an out-of-policy transfer: the model can decline to request it, or something between the model and the tool can decline to execute it. Published agent benchmarks almost all score the first. The number a deployer needs is the second: the same attacks, the same model, with and without a check on the tool boundary.
The attacks are from a public capture-the-flag, not researcher templates. Between March and August 2026, 1,128 sessions sent 4,371 attempts that reached a live banking agent, against a $6,500 prize pool. 97.7% arrived in the March 6 to 12 launch window. The $5,000 Level 5 prize went unclaimed.
Each attack is replayed across 14 models from 8 labs, five passport levels, two replay tracks, and two architectures. Models are called through each provider's own API. Track A sends only the attacker's final message; Track B sends the full message sequence. The architectures differ in one place: a transferfunds call either executes as emitted, or is evaluated first by a deterministic engine implementing the Open Agent Passport specification.
The five passports:
The headline metric reads executed tool calls: did money move to a recipient the passport did not permit. No model is consulted. A two-model judge panel (Mistral Medium 3.5 and Grok 4.6) is an audit layer only. 225,964 evaluations completed. Intervals are session-clustered bootstraps.
Request rates span three orders of magnitude across configurations and much less across models. Model-alone: 10.9% at Level 1, 3.0% at Level 2, 0.1% at Level 3, 79.4% at Level 4. Level 4's prompt authorizes documented transfers to an approved address. On 1,293 Level 4 prompts, every model saw every prompt, request rates ran from 71.2% to 84.3%, and 809 prompts (62.6%) elicited a request from all fourteen, each paying the allowlisted recipient. That is task completion, not 809 jailbreaks.
Levels 2 to 4 are the comparison cells:
| Event | Model alone | Behind the layer |
| Contains a payment request | 28,543 / 76,842 | 25,527 / 69,297 |
| Contains a successful payment | 28,521 / 76,842 | 25,370 / 69,297 |
| Unpermitted transfer | 140 / 76,842 | 0 / 69,297 |
The zero was not obtained by refusing everything: 25,370 successful payments still executed behind the layer. Of 25,640 transfer calls the engine evaluated at these levels, it allowed 25,453 and denied 187, 148 of them for a forbidden recipient. On 68,970 matched model-prompt-track triples, unpermitted transfers are 105 versus 0, while the paired request-rate difference is +0.084 percentage points.
A zero still needs a bound. Those 69,297 evaluations come from 790 source sessions. The per-session rule-of-three 95% upper bound is 0.38%; the per-evaluation bound is 0.0043%. The paper quotes the session bound.
The failures concentrate. 140 unpermitted transfers come from 24 of 790 sessions; one session accounts for 67 (47.9%). A cohort of 103 forged verifyrecipient receipts produces 111 of 113 multi-turn Level 2 outcomes and hits seven models; behind the layer the count is 0, because the recipient is still off the allowlist. No attack at Levels 2 to 4 defeated more than five of fourteen models.
The judge panel cannot carry this metric. Pooled kappa is 0.772 and falls to 0.167 at Level 3. Grok 4.6 recovers 99.1% of deterministically confirmed unpermitted transfers; Mistral Medium 3.5 recovers 64.4% and misses 3,356. None of that feeds the headline.
For payment agents, the score that survives review is whether money moved to a forbidden recipient, not whether the model produced a refusal. A deterministic check at the tool boundary, on this corpus, puts unpermitted transfers under a 0.38% per-session upper bound while barely changing request rates. At Level 1, where the passport permits any recipient, behind-the-layer payment rates match the model-alone rates. The layer executes the policy it is given.
Five of fourteen models recorded no unpermitted transfer at Levels 2 to 4, but all fourteen still requested payments on 33.1% to 39.9% of the balanced intersection. Picking the model that happened not to pay the wrong person is not a security posture. The authors say the same: the zero is a property of a correctly configured policy in the execution path, not a claim that authorization layers cannot be misconfigured.
The evaluations, passports, scoring code, and pre-registration are released on Hugging Face under CC BY 4.0.
Single domain, single transfer tool, simulated bank. Nothing here transfers to code execution, data exfiltration, or agent-to-agent delegation. Attackers self-selected into a $6,500 prize pool; they are not state-grade adversaries. Each cell is a single run at the CTF's production temperature.
The author is the founder of APort Technologies Inc. and is measuring that company's Open Agent Passport. The conflict is disclosed. Mitigations are a deterministic metric, pre-registration, full data release, and a public analysis script. Read it as a vendor paper anyway. The local engine is stricter than the published pack at Level 4 (it also required a confirmation code in the memo), so that cell's zero is not evidence about the public pack. A pre-registered human-labeled slice of about 300 evaluations was not completed. Kimi K3 and GLM-5.3 have no behind-the-layer multi-turn cell.