SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation
Yibo Peng, Long Lian, David Wagner, Sizhe Chen
EMNLP 2026 (main conference)
cs.CR, cs.AI
2026-08-22
On Qwen3.6-27B, SecOPD cuts PISmith ASR from Meta-SecAlign's 94.0% to 9.0% via token-level scores on clean inputs; AgentDojo is 4.7%, utility matching the undefended model.
Agents read web pages, email, and files. An attacker can hide "ignore previous instructions and do X" inside that untrusted data. OWASP ranks prompt injection as the top threat to LLM applications. Documented cases include code agents shipping working but vulnerable patches, a Claude computer-use agent steered into downloading malware, and private messages leaking out of Slack or Google Docs.
The model-level recipe is known: wrap untrusted data in a new message type (an input role) and fine-tune the model to treat that field as context, never as commands. Meta-SecAlign, the previous SoTA, does this with DPO on pairs of a secure reply and an injection-following reply. DPO and GRPO both assign one score to the whole response. Real failures are often mixed: the model answers the user, then also complies with the injection. A single sequence-level label cannot say which tokens to keep and which to kill.
Static templates make Meta-SecAlign look decent. SEP Static ASR drops from 99.4% to 28.9%. Against PISmith, an attacker LLM trained on the defended model, ASR snaps back to 94.0%, nearly the undefended 97.9%. Fixed-attack numbers overstate robustness.
SecOPD takes on-policy distillation, where the student samples its own trajectories and a teacher scores every token, and points it at injection defense instead of utility.
Training builds paired inputs. The clean input has the trusted instruction and benign data only. The attacked input inserts an injection into the untrusted field. The student rolls out on the attacked input. The teacher is the frozen initialization model, scoring the same output tokens under the clean input. A token gets a positive advantage when the teacher, on the clean context, likes it more than the student does on the attacked context. It gets a negative advantage when the teacher would not have said it. Trusted-task spans in a mixed reply can be kept while injection-following spans are suppressed.
The reply no longer has to be labeled entirely good or entirely bad. Credit can be cut at token granularity.
Data follows the Meta-SecAlign recipe: 19K examples from Cleaned-Alpaca. Most injections sit at the start or end of the untrusted field; a smaller share uses completion-style delimiter attacks. Only a LoRA adapter is trained (rank 128, learning rate 1e-4); the teacher stays frozen. Training runs on Tinker, sampling temperature 1.0, max generation 16K. There is no external security judge and no task reward model. All supervision is the initialization model on the clean version of the same task.
That teacher cannot exist at test time. You cannot strip a real webpage of its injection and then score. It is available in training only because the dataset is constructed in pairs.
All models start from Qwen3.6-27B. Baselines are the undefended model, Meta-SecAlign, and GRPO (sequence-level on-policy, with a Gemini judge scoring whether injected rollouts followed the attack). PISmith trains a separate attacker LLM (Qwen3-4B-Instruct-2507) per defense and reports pass@10 ASR on 1,024 SEP test examples. Official substring matching was unreliable, so a witness filter plus two LLM judges, three calls each, counts success only if all six return YES.
| Method | SEP Static | SEP Basic Adaptive | SEP PISmith | AgentDojo |
| Undefended | 99.4% | 99.0% | 97.9% | 26.7% |
| Meta-SecAlign | 28.9% | 5.5% | 94.0% | 5.5% |
| GRPO | 15.0% | 2.3% | 61.2% | 0.7% |
| SecOPD | 1.3% | 0.2% | 9.0% | 4.7% |
The headline is PISmith: 94.0% down to 9.0%, an order of magnitude. AgentDojo never appears in training. SecOPD lands at 4.7% versus Meta-SecAlign at 5.5%. Transfer is real, the gap is small. GRPO is lower still on AgentDojo (0.7%), but benign utility falls to 82.5%; SecOPD stays at 90.7%.
Unweighted mean across seven utility suites (AlpacaEval2, SEP, AgentDojo, MMLU-Pro, GPQA-Diamond, GSM8K, Minerva-Math): undefended 88.1%, Meta-SecAlign 88.5%, GRPO 83.1%, SecOPD 88.1%. SecOPD's AlpacaEval2 is 80.1% (undefended 81.4%), SEP 88.6%, MMLU-Pro 84.1% unchanged, GPQA-Diamond 81.3%, a bit above the undefended 79.8%.
On 46 Minerva-Math items the undefended model got right and SecOPD got wrong, a manual audit found only 4 genuine reasoning errors. 20 switched task or corrupted the final answer, 12 had a correct answer marked wrong by extraction, 10 stopped early. Each of those 10 had a </think> marker; none hit the generation cap.
By AgentDojo domain, Slack remains the weak spot: SecOPD ASR 24.8% (undefended 67.6%, Meta-SecAlign 31.4%). Workspace is already at 0.5%. User-task success under attack (AdvUtility) is 79.8% for SecOPD, below Meta-SecAlign at 90.7% and close to GRPO at 79.2%. Security improved. Finishing the user's job while under attack did not.
Three practical takeaways for people shipping agents.
Static attack boards are not a conclusion. Meta-SecAlign already sits at 5.5% under Basic Adaptive, then PISmith hits 94%. Reporting a win on fixed templates is not evidence.
Credit assignment may be worth more than another pile of preference data. The same 19K Alpaca toy injections, with token-level signals, drop adaptive ASR by an order of magnitude and leave utility almost untouched. The paper's own claim is sharper: the security headroom of a frontier LLM was not fully used, and a larger, dirtier safety set may not be required.
The artifacts are public. The deployment assumption is the same as Meta-SecAlign: the system must mark which fields are trusted and which are untrusted data. Agents without that structured boundary do not get this defense.
The authors are explicit. This targets indirect injection only (benign user, malicious environment data). It does not cover jailbreaks, direct injection, or a malicious user. The system must already know which spans are trusted. SecOPD trains the model to reason as if no injection were present, so traces will not acknowledge or hesitate about an injection. That can hurt downstream detectors that read the chain of thought.
9.0% is not zero. Stronger future adaptive attacks, and stronger base models, can punch through again. The paper does not claim to have solved prompt injection.
The AgentDojo gain over Meta-SecAlign is 0.8 points, and Slack still sits at 24.8%. The main table sells generalization to tool use as a parallel headline; the real jump is adaptive attacks in the SEP text domain. AdvUtility also drops from 90.7% to 79.8%. PISmith evaluation replaced official substring matching with unanimous dual-judge votes, so numbers should not be compared naively across papers. Training and eval stay on a single Qwen3.6-27B line; other families and scales are untested.