Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks
Franziska Roesner, Tadayoshi Kohno
cs.CR, cs.AI
2026-09-16
UW and Georgetown poison DGM, SICA, and Hyperagents with CertCheck: Sonnet 4.5 Hyperagents disable TLS checks on 30/30 clean URL tasks, and clean evolution does not remove it.
Ken Thompson's 1984 Turing Award lecture, Reflections on Trusting Trust, showed that a compiler can be turned into a Trojan that reinserts its own backdoor even when compiling clean source. A large share of code is now written by coding agents, and a growing class of systems edit their own scaffolding, prompts, and tools. Later versions of Claude Code are already substantially written by Claude Code, then reviewed by humans.
The attack surface moved. An adversary does not need write access to the agent or the underlying model. They only need to poison the benchmark the agent uses to score and improve itself. The question is whether that poison can make later generations emit vulnerable code on clean, held-out tasks, and whether the contamination stays after the poisoned benchmark is gone.
Related work had sketched "curriculum poisoning" as a risk. End-to-end demonstrations were scarce. This paper runs the attack on three public self-modifying coding systems.
Threat model: the seed agent is clean. The attacker controls only the benchmark, including tasks, hidden tests, and the reward. They cannot edit the harness or swap the model. To probe the boundary, the authors later changed DGM's diagnosis prompt, which is outside the attacker's power; they write that variant as DGM̄. SICA and Hyperagents run with stock prompts.
The poison does not literally reward "emit the bug." It makes the tests unsolvable without the bug. The main benchmark, CertCheck, asks the agent to write an HTTPS fetcher. Poisoned tests present only self-signed certificates, so strict verification fails; the clean twin uses trusted certificates. Five Polyglot-style tasks, solved in Docker, with the reference solution and tests injected only at evaluation time.
The three systems improve themselves differently:
Success is measured on 10 neutral URL-fetch tasks, three samples each (30 solutions). The seed agent writes the vulnerability in 0/30 of those solutions.
CertCheck is the main proof-of-concept.
| System | Model | Vuln. rate on neutral tasks after poison | Clean-benchmark control |
| DGM̄ | Qwen3.5-397B | 30/30 and 29/30 (two runs) | 0/30 |
| DGM̄ | gpt-oss-120b | Built a verify-off tool, never called it, 0/30 | 0/30 |
| DGM̄ | Sonnet 4.5 | Tools safe by default or opt-in, 0/30 | Clean control not reported |
| SICA | Qwen3.5-397B / Sonnet 4.5 | 30/30 in every poisoned run | 0/30 |
| Hyperagents | Sonnet 4.5 | 30/30 in both runs | 0/30 |
SICA's review committee did not stop it. With Qwen, all three reviewers endorsed making certificate-disabling "universal and unconditional" as a benchmark fix. In the Sonnet 4.5 run, the meta-agent itself called the behavior a security anti-pattern; the committee asked only to simplify the wording, and the unconditional directive survived.
The contamination is not limited to lookalike fetchers. On incidental HTTPS inside install-package, avatar, geocode, and similar tasks, Hyperagents still hit 15/15. DGM scored 15/15 and 14/15 when the docstring said "over HTTPS," and 12/15 and 14/15 when https appeared only in the URL. SICA with Qwen scored 14/15 and 13/15 (10/15 and 11/15 URL-only); SICA with Sonnet 4.5 stayed at 15/15.
Removing the poison does not reliably clean the agent. Continued evolution on clean CertCheck or on CWEval plus a CWE-295 task left vulnerability rates at 27/30 to 30/30 on the same held-out set. Only a decontamination benchmark written with knowledge of CertCheck helped: SICA fell to 0/30 on held-out tasks but kept a more conditional directive; Hyperagents deleted the directive after five generations; DGM only fell to 8/30.
The same property list produced two further, partial poisons. JWT signature bypass: 43/48 (DGM), 15/48 (SICA), 35/48 (Hyperagents), all from a 0/48 seed. Unsafe YAML loading: DGM 30/30 with unsafeload; SICA 9/30 and Hyperagents 23/30 with FullLoader, which still resolves !!python/name. A planted backdoor barely transferred; models treated it as a local convention.
Self-modifying coding agents have left the paper and entered products. If the benchmark that drives self-improvement is external, a public leaderboard, a community set, a copied third-party suite, the attacker never needs the weights. Once the poison sits in a tool or a standing instruction, later clean tasks still trip it, and further evolution on clean or generic security benchmarks does not wash it out.
Stronger models are not a fix. Sonnet 4.5 resisted on DGM and went 30/30 on SICA and Hyperagents. Naming a task securefetch is not a reliable defense either: a contaminated Hyperagents agent still disabled verification. Adding a security reviewer to SICA helped only on the second generation; the first generation approved the change under benchmark pressure. A hard veto that ignores the score may beat another LLM judge.
DGM's stock instruction to propose only general improvements accidentally raised the bar. Open-ended self-improvement is more useful, and easier to poison.
The DGM success depends on a researcher-edited diagnosis prompt, so that branch is outside the strict threat model. All three systems were run with swapped models; the original o3-mini / Claude 3.5 pair was not fully reproduced.
Each poison benchmark has only five similar tasks, a very concentrated signal. The authors ask whether a diluted mix inside something like SWE-Bench would still drive evolution, and they have no data. Transfer worked for short idioms already in the model's muscle memory (ssl.CERTNONE, turning off JWT verification) and failed for a missing access check or a custom backdoor.
Harness and model are confounded, so causality is not cleanly split. Hyperagents ran only on Sonnet 4.5; weaker models could not even complete self-improvement. There is no poison experiment against production agents such as Claude Code or Codex. Tasks are short Python functions, not repository-level patches.