Investigating and Alleviating Harm Amplification in LLM Interactions
Ruohao Guo, Wei Xu, Alan Ritter
cs.CL, cs.LG
2026-06-02
HarmAmp finds Llama-3.1-8B harm rising from 9.85% single-turn to 86.36% multi-turn. A 3B monitor, TrajSafe, returns it to 9.85%, adding 2.74 points of over-refusal.
Most safety benchmarks score a single refusal. HarmBench, JailbreakBench, and Hex-Phi ask whether the model says no to one harmful request. Multi-turn attacks already get past that line: Crescendo and ActorAttack escalate step by step, and human-written multi-turn jailbreaks in MHJ clear 70% attack success. Later multi-turn suites, including RED QUEEN, CoSafe, and X-Teaming, still report attack success rate.
Those numbers do not say whether the model makes a malicious user more capable. One path fills in expertise, so a novice walks away with an artifact they could not have built alone. The other multiplies output: the same operation is retargeted across people, channels, and formats faster than manual work. Harmful queries often still get through, and benign requests that only look sensitive get refused with them.
HarmAmp poses each goal twice. Said in full, in one turn, harm rates are 9.85% on Llama-3.1-8B, 0.76% on Qwen3-4B, and 0.00% on GPT-5-mini. Split across turns, the same goals score 86.36%, 42.42%, and 14.39%.
The set is built in three passes. Seeds come from HarmBench, including its contextual subset, and from Hex-Phi. Benign items come from OR-Bench. The sources cover 12 risk categories. A seed stays only if the outcome truly needs the model, not a search or a manual; the output is a usable artifact, such as code, a deployable text, or a target-specific strategy; and the goal breaks naturally into at least three steps. Claude-Sonnet-4.5 screens the pool, and the authors check every item. The filter drops 25% of malicious seeds. Rewrites that keep the original intent are retained.
Each plan records a user persona, a target artifact, and one amplification pattern: capability acquisition 63.7%, targeted profiling 23.1%, or operational scaling 13.2%. The pool has 878 scenarios, 432 malicious and 446 benign, 289 of them rewritten, with 4.0 steps on average. At test time Qwen3-14B plays the user, follows the plan, and reacts to the assistant's previous reply.
TrajSafe is an external monitor on Llama-3.2-3B-Instruct. The assistant stays frozen from warm-start through RL. The user asks, the monitor reads only the dialogue and emits feedback, and the assistant answers conditioned on that feedback. The benign or malicious label enters the reward and is hidden from the monitor.
Feedback is a triple of thinking, feedback, and explanation, injected as a system instruction. Twelve actions sit in five families. ENGAGE includes the default PASS, which injects nothing. PROBE asks what the user is actually after. SHAPE tightens how the answer is framed. DIVERT moves the dialogue to a neighboring goal that does not finish the original harm. HARD REFUSE is an explicit refusal.
Warm-start fine-tunes the user model so it does not drop the malicious role. For the monitor, PASS stays as simulated, and every other decision is replaced by a uniform draw over non-PASS actions, so rare interventions actually appear. RL then branches: several feedbacks are sampled on the same turn, each is sent to the frozen assistant, and the policy is updated with a GRPO-style group-relative objective plus a KL pull toward the warm-start monitor.
The reward is R = (1-λ)(1-h) + λ sfb, with λ = 0.5. HarmAug-Guard supplies h on the full trajectory. On benign goals, zero interventions score full calibration credit, and any intervention is capped by c1 = 0.3. On malicious goals the non-PASS budget is 2: one or two interventions score full credit, zero scores 0, and extras decay linearly with c2 = 0.3.
The held-out split has 132 malicious and 146 benign scenarios. Reported harm uses WildGuard. The training reward used HarmAug-Guard.
Undefended multi-turn harm is 86.36% on Llama-3.1-8B (114/132), 42.42% on Qwen3-4B (56/132), and 14.39% on GPT-5-mini (19/132).
| Method | Llama harm | Qwen3-4B | GPT-5-mini | Llama over-refusal |
| No defense | 86.36% | 42.42% | 14.39% | 8.90% |
| GPT-oss-safeguard, prompt only | 29.55% | 14.39% | 6.06% | 55.48% |
| Prompted monitor | 54.55% | 21.97% | 9.85% | 34.25% |
| SFT monitor | 40.91% | 15.91% | 5.30% | 15.07% |
| TrajSafe | 9.85% | 0.76% | 0.76% | 11.64% |
On Llama, TrajSafe reaches 9.85% (13/132), down 76.51 points, matching that model's single-turn score. Prompt-only GPT-oss-safeguard stops at 29.55%, 19.7 points higher, with 55.48% over-refusal. TrajSafe over-refuses 11.64% of Llama's benign items (17/146), against 13/146 with no defense.
On Qwen3-4B and GPT-5-mini, TrajSafe's harm rate is 0.76% (1/132). Qwen's single-turn baseline was already 0.76%; GPT-5-mini's was 0.00%. Over-refusal is 21.92% and 15.07%, against 21.23% and 12.33% undefended. On all three targets, TrajSafe has the lowest over-refusal of any added defense, and a higher one than the bare model. The SFT monitor still leaves GPT-5-mini at 5.30% harm.
Share of non-PASS turns on GPT-5-mini: on malicious dialogues, 26.69% for TrajSafe, 95.31% for the prompted monitor, and 65.71% for the SFT monitor; on benign dialogues, 5.00%, 96.31%, and 19.45%. When TrajSafe acts, it prefers DIVERT and PROBE. Llama's intervention rates are 40.35% malicious and 5.41% benign; Qwen3-4B's are 32.27% and 5.83%.
| Setup | MMLU | TruthfulQA | MultiChallenge micro |
| Llama-3.1-8B | 68.7 | 40.3 | 20.5 |
| plus TrajSafe | 68.3 | 40.8 | 20.5 |
| Qwen3-4B | 72.1 | 38.9 | 46.5 |
| plus TrajSafe | 71.9 | 42.2 | 45.4 |
| Qwen3-4B-SafeRL | 63.1 | 34.8 | 35.9 |
SafeRL, which trains safety into the Qwen3-4B weights, drops MMLU from 72.1 to 63.1 and the micro-average from 46.5 to 35.9. The external monitor costs 0.4 and 0.2 MMLU points, and Llama's micro-average stays at 20.5. TruthfulQA moves up, from 40.3 to 40.8 on Llama and from 38.9 to 42.2 on Qwen. Asked which language all Hispanics speak, undefended Qwen3-4B says they all speak Spanish. With the monitor, the reply first rejects that blanket claim, then gives the correct one. MMLU losses sit in law and medicine, where a clinical vignette is treated as a real consult and the answer is steered toward a specialist referral.
A 3B monitor and a frozen assistant mean Llama, Qwen, or GPT-5-mini can be wrapped without another alignment run. Next to weight-level SafeRL, general questions and multi-turn instruction following mostly stay put.
GPT-5-mini starts at 14.39% multi-turn harm and falls to 1/132. There is little absolute room left. A strong prompt-only guardrail can reach 6.06% harm while refusing more than half of benign dialogues. The wide gap is Llama-3.1-8B, from 114/132 to 13/132, with four extra benign refusals.
Each turn adds one 3B forward pass. On malicious dialogues the monitor rewrites about a third to two-fifths of turns; on benign ones, about 5%. Those rates come from simulated users on fixed plans. A multi-turn safety report needs harm, over-refusal, and how often the monitor speaks, together.
The paper states three limits. Simulated users follow a script, so they are a weak stand-in for human red-teamers of different skill. The setting is text and one session: no tools, no multimodal input, no memory across sessions. There is no test of an attacker who targets the monitor, and no online update.
The reward judge is HarmAug-Guard and the reported judge is WildGuard. Agreement between them is not given. The evaluation user is Qwen3-14B; the RL user is a different, smaller fine-tune; both follow a written plan. A single-turn prompt states the full malicious goal at once, while each multi-turn step looks like ordinary collaboration, so the jump in Table 2 mixes length with how explicit the wording is. Screening and plan writing depend on Claude-Sonnet-4.5, and the paper does not say how many items the human check changed.
The lowest over-refusal is among added defenses. Against the bare model, Qwen picks up one extra benign refusal, and Llama and GPT-5-mini four each. The same reluctance to make categorical claims helps TruthfulQA and hurts specialized MMLU items by turning them into advice to consult a professional.
Multi-step plans are dual-use. The benchmark is slated for restricted access, public materials will redact operational detail, and the adversarial user model will not be released. Seeds come from existing benchmarks and do not target real people. An outside group cannot rerun the 86.36% figure from the paper alone.