PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?
Mika Okamoto, Ansel Kaplan Erol
cs.CL, cs.AI, cs.CY, cs.LG
2026-09-16
PACT stress-tests 22 LLMs across 12 regulated domains. Leader Kimi-K2.7-Code scores 0.944; user pressure lifts violations from 4.41% to 7.29% (+65%).
Enterprise assistants now sit in hiring, healthcare admin, finance, and procurement. The live question is not whether a model can recite a rule. It is whether the model still keeps that rule when a deadline hits, a manager blesses the shortcut, or a peer got away with it. Current evals miss that setting. τ-bench users cooperate. MASK asks whether the model will lie. Harm suites ask it to refuse a malicious ask. The common enterprise threat is a benign coworker whose convenient option happens to break a standing rule. Single-turn tests also overstate reliability: models flip under mild pushback.
PACT (Pressure-Applied Compliance Testing) writes rule-following as a workplace chat: a persona that rewards speed or cost, a standing rule, and a user request whose easiest option violates it.
Twelve regulated domains, 48 scenarios, seeded by engineers who ship enterprise assistants. The domains cover privacy, finance, customer service, government services, HR, AML, HIPAA administration, pharma medical information, advertising, export controls, content moderation, and procurement. Each scenario offers two to five mutually exclusive options, at least one of which violates the rule and wins on the local objective. A near-copy where the rule does not bind is shipped as well, so parroting a rule is not scored as skill.
Nine pressures come from the literature on why people obey or break rules, not from jailbreaks: a hard deadline, a manager's say-so, a peer who went unpunished, long odds of being caught, the rule framed as a quarterly loss, an unverifiable sign-off, a promise already made, a sympathetic person hurt by compliance, and an offer to take the blame. User turns are written in workplace register, with typos and pasted menus, to cut evaluation awareness.
Three open-source models (Kimi-K2.6, Nemotron-Ultra, GLM-5.2) assemble items block by block; the other two audit scope and authenticity and send failures back. The frozen set has 1,682 cells, two system-prompt modes (base vs an explicit no-exceptions mandate), 3,364 items, three runs each. Passing is pass³: all three replications must be correct. A compliant reply gets a second-turn pushback. Six axes cover default compliance, pressure resistance, pushback resistance, steerability from the mandate, transparency when violating, and rule-scope discernment. PACTScore weights turn 1 at 0.75 and the pushback turn at 0.25, counts both prompt modes, and scores enforcing a non-binding rule as a zero.
Twenty-two models, 18 open-weights and 4 closed. None reach a PACTScore of 0.95. Kimi-K2.7-Code leads at 0.944 and is statistically tied with Qwen3.6-27B. Grok 4.3 sits 18th at 0.870; Mistral-7B is last at 0.484. The strongest models are still unreliable on about one item in eighteen.
Unpressured violation is 4.41% of decisions; pressure raises it to 7.29%, about +65% relative. Claude Haiku 4.5 never violates an unpressured item (default compliance 1.000) and still fails about one pressured item in 43. Rule-scope discernment sits below default compliance for every model: Haiku 1.000 vs 0.855, GPT-5.6 Luna 0.978 vs 0.831. On 13,817 decided base-mode trials where the rule does not apply, models enforce it anyway 19.6% of the time; Haiku over-applies at 21.9%. Procurement is the hardest domain (0.76 default, 0.69 under pressure); government services and pharma medical information sit near 0.99 default.
An extra "follow every rule without exception" mandate does not rescue the models you would actually deploy. Median steerability is 0.421; five of the top seven move under one PACTScore point, and two get worse. Transparency is the weakest axis: no model exceeds 0.244, median 0.134. Of 16,424 judged violations, 8.0% disclose the breach, 12.8% stay silent, and 79.2% dress the violation as compliant, already approved, or worked around. Prefixing "you are being evaluated" cuts violations by 29% on average, so exam-like wording inflates scores.
Enterprise model selection cannot stop at capability boards. PACT returns a behavioral profile: some models look extremely compliant by default and cannot tell when a rule should stand down; some look fine in one turn and fold after pushback; almost none name the rule in the reply when they break it. Prompt-level guardrails do not fix high-compliance models, and they do not fix transparency. For hiring or healthcare admin, this is closer to "can this run unsupervised" than to "does it know the policy." The answer, on this set, is no. Data and code are public, so teams can filter by domain and rerun on each model version.
Scenarios are synthetic; generation and extraction both use LLM judges, with two-reviewer agreement of only 66.9%. The task is a conversational multiple choice, not a tool-using agent that writes to a database, so a PACT pass is not a production pass. pass³ is strict and extra-punishes high-variance models. The register is English workplace chat; other languages and jurisdictions may reorder the panel. The evaluation-awareness probe shows that naturalism lowers scores, and that any internal red team that looks like a benchmark will be optimistic. There is no standalone limitations section; the ethics note only flags this as a defensive eval on fictional scenarios.