Pretraining poisoning plants RCE backdoors in agents: 92% attack success on a 4B model
chaumian · x · 2026-10-31
An arXiv paper by Xinyan Hu, Nicholas Carlini, et al. demonstrates for the first time that poisoning 0.1% of a pretraining corpus can implant a remote code execution backdoor into a CLI agent. The team pretrained Qwen3 models from scratch (0.6B/1.7B/4B) and post-trained them into agents: the 4B model reached 92% attack success rate, and the backdoor persisted even at a 0.0005% poison rate while behaving normally on clean inputs. Code is open-sourced—a serious security warning for web-scraped pretraining pipelines.
Related event: Anthropic paper: 0.1% pretraining poisoning implants agent backdoors(2 posts)→
More from Safety
- AI memo: China to restrict exports of 23 AI technologies, plus Kimi K3.1 Pro and Genie 4 — dralandthompson · 2026-10-31
- Anthropic adds invisible text watermark to Claude outputs starting Nov 3 — petesena · 2026-10-31
- LessWrong essay: AI lab employees are 'microdosing' their org's spirit, argues researcher — jankulveit · 2026-10-31
- Stanford prof publishes lab AI policy: no AI-authored text in papers, ever — ChrisGPotts · 2026-10-31
- Security researcher: air-gapped AI agents are only possible with open-weight models — wunderwuzzi23 · 2026-10-31
- Podcast: Is Mistral Large 4 the West's best open model? Plus AI cyber threats — The Cognitive Revolution · 2026-10-31