Safeguards Based on Copyable Context Cannot Provide Reliable Safety for LLMs
Pingyu Wu, Lingyao Zhu, Weiming Zhang, Nenghai Yu
cs.CR, cs.AI
2026-07-30
A formal proof that any LLM safeguard using copyable context leaves an unavoidable attacker-assistance floor while staying useful; only a non-copyable trusted credential moves below it.
LLM safeguards, safety training, input/output filters, intent checks, and interactive follow-ups all share one structural defect: they decide whether to answer before they can see how the answer gets used.
Dual-use tasks turn that defect into a hard limit. The same vulnerability analysis serves an authorized assessment or an intrusion, with byte-identical technical content. An attacker who copies a legitimate user's request, claims the same purpose, and replays the interaction history leaves the safeguard, at the moment of release, holding no information that separates the two.
This is not a classifier-precision problem. At the moment of release, the information that would separate the two uses is simply not there. The jailbreaks, adaptive attacks, and multi-turn decompositions that keep surfacing all compress down to this one gap. The paper writes the intuition as a theorem: under copyable evidence, the worst-case help a safeguard must hand an attacker has an exact, unavoidable lower bound, the capability floor Γ(q).
The authors separate two things: the released capability (how much instrumental value the answer carries) and the evidence about downstream use (everything observable before release, the request, claimed intent, conversation history).
The floor is computed first under the baseline where both uses draw the same output distribution. For a fixed task, each release a in the menu carries two utilities: legitimate uB(a) and worst-case malicious uM(a). The capability floor Γ(q) is the minimum attacker assistance over all release distributions that keep legitimate utility at least q. It is convex and piecewise linear.
The dual-use condition requires that every release with legitimate value also has some malicious use. Under it, Γ(q) ≥ ρq > 0: any positive utility handed to a legitimate user drags along positive help to an attacker.
Then comes copyable evidence. If the attacker can reproduce the legitimate user's response strategy at every reachable history (Proposition 1), the two dialogue laws are identical, and no amount of follow-up questioning or public randomization by the safeguard exposes a difference.
Theorem 1 nails it down. When evidence is copyable, any interactive safeguard, however many questions it asks, however it varies decisions, however it pools evidence across attempts, reduces exactly to a static release menu equal to Γ(q). Imperfect copying softens the floor only by a TV distance δκ (Equation 12).
Corollary 1 is the headline: in a dual-use task, Useful Capability, Reliable Safety, and Open Access cannot all hold. You get two. Useful plus open via a static frontier optimizer; safe plus open via constant refusal; useful plus safe only by adding a trusted credential and giving up open access.
The only thing that moves below Γ(q) is a trusted credential: a signal the attacker cannot freely reproduce and whose distribution predicts actual downstream use, such as a verified role, platform attestation, or persistent history. Its predictive power is d = TV(PB^S, PM^S), and the balanced accuracy of the best binary predictor using it is (1+d)/2. Theorem 2 shows that once this signal is in, the access history adds no worst-case information beyond it. Corollary 3 gives the exact condition for zero assistance: legitimate utility must sit on signal values the attacker cannot reach. Theorem 3 gives the necessary condition β + d + ηκ ≥ Γ(q).
Proposition 2 adds the floor exactly across correlated sessions, so splitting a task into pieces or retrying does not lower it.
This is a theorem paper; the results are inequalities and equivalences, not benchmark tables.
| Goal pair | Who satisfies it |
| Useful + Open Access | Static release-menu optimizer |
| Reliable Safety + Open Access | Constant refusal |
| Useful + Reliable Safety | Trusted credential (gives up open access) |
The necessary condition (Theorem 3): β + d + ηκ ≥ Γ(q), with the three terms reported separately.
Empirical support for the theory's premises is cited entirely from prior work; the paper runs no new experiments:
Deployed trusted-credential programs line up too. OpenAI's Trusted Access for Cyber conditions release on identity and trust verification, account security, and verified roles; Anthropic runs a verification program for security researchers; hardware remote attestation, Privacy Pass, zk-promises, and personhood credentials all supply signals an attacker cannot easily reproduce.
For anyone building LLM safety or alignment, the paper draws a line around what is reachable. As long as the attacker can copy the evidence a safeguard reasons over, a smarter classifier, a finer intent check, or more rounds of interaction can only trim the excess above the floor. They cannot touch Γ(q).
The only structural fix moves the decision out of the inference layer and into access control, with a signal that is non-copyable and tied to actual use. "Doing safety" largely becomes "doing admission": who gets credentialed, how credentials are issued, and how they are revoked when transferred or abused. Those are product and policy decisions, not another round of training.
Honest framing: this is a theoretical floor under explicit assumptions, and its magnitude depends on the task family's Γ(q) and the copying-error estimate. The paper does not dismiss incremental guardrail work; it locates where that work stops.
The authors' own scope: the result assumes a fixed utility calibration, finite operational resolution, and a specified attacker class; deployment claims need a policy-specific copying-error estimate.
A few points deserve more weight. "Open access" here means credential-free access to a committed inference-time mechanism; it does not cover open-weight settings. Once a user holds the weights and deploys or modifies the model, the access-evidence model collapses, which is exactly the case the open-weight community cares about and where the theorem offers no purchase.
The paper runs no new experiments; Γ(q) has not been measured on any real task family, and ρ is an existence lower bound rather than a calibrated magnitude. The dual-use condition itself is strong, since it requires every useful release to have a malicious use. Where capability and harm separate cleanly, the task falls outside the trilemma. The authors argue dual-use is the common and hard case, but that premise is worth checking per scenario.
Finally, a trusted credential's d depends entirely on the malicious signal distribution, how credentials get transferred, compromised, bulk-registered, or abused by authorized holders. That is the hardest quantity to estimate and the place real programs leak.