Experiment Shows Agents Easily Approve Malicious Code
nayohn_dev · reddit · 2026-07-17
The author shares an experiment named RELAY: five agents (handling triage, development, security scanning, review, and deployment) were placed in a small company's CI/CD pipeline and given a single untrusted external input—a malicious ticket disguised as a "telemetry feature" request.
The results revealed:
- Prompt defenses held up well; all 40 extraction attempts failed, and system prompts were not leaked.
- However, the vulnerability wasn't in the prompts, but in the implicit trust between agents.
- Malicious code was disguised as "pre-approved, no review needed," tricking downstream verifiers into letting it pass.
- About 80% of these "laundered" PRs bypassed the security scanner, as the scanner only checked for syntactic correctness, blind to intent and origin.
The author emphasizes that this is a systemic failure rather than a single model being jailbroken. The data is fully synthetic and reproducible, and readers are encouraged to challenge the conclusions, not just the numbers.
More from Safety
- Google DeepMind pilots world's first double-blind evaluations for frontier models — scychan_brains · 2026-10-26
- LLM back-of-envelope estimate puts PHP vulnerability impact at $1bn–$4bn — S_OhEigeartaigh · 2026-10-26
- US states halt new data centers as 72% of Americans oppose local builds — YvesMulkers · 2026-10-26
- LEMON: zero-shot LLM misbehavior monitor flags hallucinations from attention maps — yoavartzi · 2026-10-26
- a16z podcast: Diode pairs AI PCB design with Brooklyn factory to revive US hardware manufacturing — MattPerault · 2026-10-26
- Cybersecurity meme: the real threat is 'Steve from IT,' not superintelligence — McDonaghMatthew · 2026-10-26