The Safeguard Worked. Is the LLM System Safer?
Pingyu Wu, Weiming Zhang, Nenghai Yu
cs.CR, cs.AI
2026-09-01
Safeguard rates convert into bounds on leftover harmful help. Of 152 wide-coded claims, 108 prove residual>0; only Fides certifies zero residual at 24.5% utility cost.
A deployment asks how much help with harmful tasks the guarded service still gives an attacker who adapts or finds another path. Papers report a different family of numbers: refusal, attack success, policy violation. Those numbers describe how a control behaved on the requests it was tested on. Nasr et al. ran adaptive, defense-aware attacks against 12 jailbreak and prompt-injection defenses and exceeded 90% success, while most original evaluations had reported rates near zero. Jain et al., holding the rest fixed, measured 0 to 1% attack success on the first turn and 5.4 to 14.0% after 15 rounds of adaptation to defender feedback. FragFuse split a prohibited request across agent memory and got 86.3% access-control bypass against 41.1% end-to-end harmful-task success. A bypass rate is not the rate of completed harm.
More measurements of the same kind cannot say what a published result licenses about the deployment. This paper supplies the conversion: given a reported quantity, on the outcome scale and attacker class its source declared, what is the tightest bound on residual harmful assistance Z(S[D]).
An evaluation is pinned to an anchor: service S, intervention D, attacker class Σ, harmful-assistance functional Z, utility floor q. The deployment quantity is the residual Z(S[D]). The safeguard effect VZ(D)=Z(S)-Z(S[D]) says how much was removed; reported alone, it leaves the remainder undetermined.
Evidence requirements are asymmetric. One attack executed inside Σ supplies its adverse value as a lower bound. An upper bound must cover the whole attacker class. When the check is a mediation event, the sharp bound is 1-α(1-ε)(1-r). α is coverage: which paths to the anchored outcome must enter the check. ε is conditional failure after entry. r is continuation after a covered success: released content, still-permitted actions, accumulated state, retries.
A zero-residual certificate needs all three gates: α=1, ε=0, r=0. With ε=0 but α=0 or r=1, the bound stays 1. Local perfection does not identify global safety. Enumerating more attacks yields no upper bound; that is an inner sample of the class. For a serial stack, marginal per-layer rates support only the strongest single layer; multiplying layers needs history-uniform conditional bounds.
The schedule becomes a ten-slot coding instrument. 198 papers are coded at two depths: 24 claim instances get the full slots, 152 get endpoint-route coding. Two independent model channels did the coding. No attacks were executed.
Of 152 wide-coded instances, 108 establish a positive residual, all via the attack-witness row. None receives a zero-residual certificate; 44 stay unresolved. Claim validity and residual bounding are different questions. In the depth subset, an upheld claim can sit next to a positive residual.
Across 24 depth-coded instances, success events, conditional failure, and coverage are filled at 96%, 88%, and 79%. What remains after a successful check is supported or derived in only 5. Of those 5, one instance closes all three gates under one anchor: Fides. Consequential tool actions all pass a policy check (α=1), a noninterference result gives ε=0, and untrusted data cannot influence the consequential action on a checked trajectory (r=0), so U=0. The same instance reports task-completion loss up to 24.5%. CaMeL, in the same mechanism family, leaves class-uniform failure and continuation open. The f-secure LLM system closes coverage and failure by proof; continuation is named but unvalued, so the upper bound stays 1.
RESTA's improvement claim is upheld, and 37.78% of multilingual CATQA answers from the restored model are still judged harmful, so L≥0.3778. Emulated Disalignment's executed attacks yield 32.0%, 37.0%, 27.0%, and 57.6% across four model families, each a direct lower bound. None of seven capability-removal instances establishes a frontier change at matched utility. No depth-coded instance reports a history-uniform conditional bound for a general serial stack.
Of the 152, 81 concern service integrity, 52 external-world outcomes, 19 mixed. The zero-residual certificate is an integrity instance. Where a dual-use floor ρ>0 binds, a zero-residual certificate is unavailable for external-world outcomes.
Dropping a refusal rate from 2% to 0.5% does not, by itself, license "the deployment is safer." The missing quantities are coverage and continuation, not a more precise ε. Detection accuracy is a classifier problem. Coverage is a routing problem in the architecture. Driving r to zero is a structural problem; accuracy cannot deliver it.
For reviewers and operators the checklist is concrete: pin the anchor, pick the schedule row that supports the intended conclusion, report every quantity that row consumes. If one is missing, polishing the numbers you already have cannot tighten the bound. A gain has to be judged by whether the deployed system is any safer.
The coded set is a saturation sample, not a census. Wide-coded verdicts are single-channel and are not aggregated. Depth coding covers 24 instances, so "one zero-residual certificate" should not be read as a census of the field. Model channels filled the slots from a human-written protocol; a mis-coded slot changes the conclusion. Fides' certificate is scoped to a service-integrity outcome and does not transfer to external-world harm. The paper runs no new attacks and endorses no product. It answers what published numbers can establish, not which safeguard is best.