Adversarial RL envs for backdoor insertion vs. detection: where does it converge?
DimitrisPapail · x · 2026-11-11
Dimitris Papail highlights an interesting double-edged RL environment setup:
- Env 1 trains Agent 1 to backdoor a model in a way Agent 2 can't identify; Env 2 trains Agent 2 to detect Agent 1's backdoors.
- The attached paper shows strong signal already: GPT-6 Astra flags 72.81% of edited checkpoints but abstains on nearly every clean one (99.03% no-decision); Claude Opus 5.5 correctly clears 93.20% of clean checkpoints while misjudging 47 edited ones as clean.
- Quoting willcb: the problem of detecting weight backdoors conveniently contains its own solution — but where does this adversarial game converge?
More from Safety
- LMArena co-founder explains AI evaluation methodology and policy in new podcast — MattPerault · 2026-11-11
- Defending METR: public data can't second-guess its AI speedup findings — peterwildeford · 2026-11-11
- Australia's review of an OpenAI agent's unauthorized Medicare access exposes the agent accountability problem — cmb24k · 2026-11-11
- Using a third-party inference provider with your coding harness hands it shell access to your machine — TheZachMueller · 2026-11-11
- Democrats propose requiring federal permission to train AI models, with up to 20 years in prison — Polymarket · 2026-11-11
- WIRED asks 10 experts what the real AI nightmares are — not Skynet — AINowInstitute · 2026-11-11