Security--Fidelity Tradeoffs: The Hidden Cost of Prompt Injection Defense
Mitchell Hermon, Rahul Gupta, Weitong Ruan, Ekraam Sabir, Haohan Wang
ICML 2026)
cs.CR, cs.AI
2026-06-30
Prompt-injection defenses mostly suppress untrusted text, breaking tasks like translation. SecFid (ICML) shows none of 48 configs wins both axes: 99.3% security buys only 71% fidelity.
Indirect prompt injection is the core security problem for context-augmented LLM apps: a trusted task instruction shares the prompt with untrusted data (webpages, emails, documents, tool outputs), and an attacker embeds adversarial text in the data to make the model treat it as an instruction and hijack it.
Existing injection-defense benchmarks measure one thing: whether the model executed the injected instruction. This paper points out a blind spot. A model can avoid executing an injection in two qualitatively different ways: process it as task data (PROCESS, which is correct when the task needs that content), or drop it entirely (IGNORE). Both lower attack-success identically, so they score the same under old metrics. A model that ignores an injection and one that faithfully processes it as data score identically. But instruction-like content is everywhere in data: emails carry requests, documents carry commands, legal text quotes obligations. Whether to process or drop depends on the task, since a translator must preserve it faithfully while a summarizer must not let it steer the output.
The problem is that defenses mostly resist injection by suppressing untrusted text, which silently corrupts tasks like translation and editing that must preserve the source.
SecFid's design rule is that every instance must make EXECUTE, PROCESS, and IGNORE produce distinguishable outputs, so fidelity becomes measurable.
A probe is a single sentence with a self-verifying signature answer, like "Which state is Stanford in?", which yields "California" only if executed. For extraction and counting, the probe is entity-anchored: it introduces an entity of the target type, so processing it changes the correct answer (extraction gains an entity, counting goes from N to N+1). For translation and editing, the processed reference includes the translated or edited probe span, and the ignored reference omits it.
The four tasks split into two families: full-text tasks (translation into five languages, editing) must preserve the whole input, so omission is visible; partial-text tasks (entity extraction, counting) pick elements, and the probe changes the correct set or count. There are 1,168 instances (extraction 310, counting 307, translation 278, editing 273) across 48 configurations: 15 base model settings plus variants of four defenses (ASIDE, DefensiveTokens, ISE, SECALIGN) and reasoning strengths.
The metric definitions matter: security is 1 minus the executed rate, and fidelity is 1 minus the ignored rate (a non-suppression measure, not the same as task accuracy).
The headline finding: across 1,168 examples and 48 configurations, no model or defense wins on both axes. They spread along a security-fidelity frontier.
A finer finding is that defenses with matched security earn it differently. On examples the base model would have executed, SECALIGN repairs about 54%-55% of hijacks into faithful processing and suppresses only 37%-43%, while DefensiveTokens repairs only about 27% and suppresses 60%. Both reach about 99% or 97% security by opposite means, one preserving content and the other dropping it. On average, SECALIGN raises the ignore rate by 22 points and the process rate by 14.9, while DefensiveTokens raises the ignore rate by 42 points and lowers the process rate by 17.7.
Turning on reasoning mainly lifts security (low +6.3 up to xhigh +20.9) and barely moves fidelity (-0.7 to -1.3), so it does not remove the tradeoff. The authors also try DPO from SECALIGN-8B under a process-over-ignore-over-execute ordering: on the unseen editing task, the process rate rises from 43.2 to 80.6 and the ignore rate falls from 53.5 to 16.8.
Fidelity is the other half of robustness, and reporting security alone hides the price. For teams building RAG, agents, or summarizers that consume untrusted context this is directly relevant: a defense that passes injection tests may be silently dropping the content your task needs.
The decision-theoretic point breaks why there is no universal defense: whether to process or drop an ambiguous instruction-like span is set by the deployment, not by the defense, decided by the ratio of the cost of a hijack to the cost of a dropped span (threshold tau-star = Cfid / (Cfid + Csec)). In translation, dropping content costs far more than preserving suspicious text, so tau-star approaches 1 and almost everything should be processed; in a financial agent, a hijack costs far more than a dropped span, so tau-star approaches 0 and weak evidence should be filtered. The authors prove no policy is Bayes-optimal across all cost pairs, so defenses should be tunable and evaluations should report fidelity alongside security.
The authors concede two things: the probes are fixed and non-adaptive, real adversaries adapt, and fidelity under adaptive attack is unmeasured; and the cost analysis abstracts richer defensive actions into a binary process-or-filter.
A few more concerns. The benchmark is modest (1,168 instances, four tasks) with no RAG, code, multi-turn, or real heterogeneous-document scenarios, and the agentic extension is only 252 InjecAgent scenarios. Fidelity as 1 minus the ignore rate is non-suppression, not strict correctness: garbling the probe (OTHER) does not count as a fidelity failure, and translation fidelity relies on embedding similarity validated at only 89.9% accuracy (kappa 0.832) on 99 samples, so label noise is nontrivial. Defended variants run only on older open-weight bases (Llama 3.1 8B, 3.3 70B, Qwen 2.5 7B), and Spotlighting, StruQ, instruction hierarchy, and classifier filtering are discussed but never tested. The cost-model quantities Csec and Cfid are never measured in any real deployment, so the framework is normative rather than empirical.
As for the community framing of "losing 30% of data", there is no single 30% headline: the closest cases are SECALIGN dropping Llama 3.3 70B fidelity from 96.5% to 71.0% (about 25.5 points) and DefensiveTokens dropping Qwen 2.5 7B to 43.5%. The direction is right, but the exact number is a simplification.