Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression
T. Bauer, W. P. Kegelmeyer, E. Begoli, A. Sadovnik, T. Emerson, C. Corley, N. Generous, J. Moore, B. Bartoldson, R. Goldhan, M. Goldman, M. Greaves, M. J. D. Vermeer, B. MacLennan, D. Schulker, N. VanHoudnos, J. Bansemer, Y. Bengio
cs.CY, cs.AI, cs.HC
2026-09-03
US national labs and Bengio map rogue-AI risk to five staged observables, skipping the probability debate. Persistence is already past the line; execution stays a subjective call.
Alignment research has spent years arguing about whether a superhuman system could slip human control. Operators who actually have to watch deployed models need a different artifact: a list of behaviors worth instrumenting. On 7-8 January 2025, six US Federally Funded Research and Development Centers ran a small workshop at Los Alamos. Sandia, PNNL, LANL, LLNL, Oak Ridge, and CMU's Software Engineering Institute each sent two people. Eleven outside invitees joined, including Yoshua Bengio, Melanie Mitchell, and CSET's John Bansemer. This paper is the write-up.
The authors refuse to wait for a settlement of the probability debate. They ask only: if catastrophic rogue-AI risk were to materialize, which behavioral conditions would have to hold, and which of those conditions leave a signal.
Five categories sit in a rough sequence. Later ones presuppose earlier ones.
Motivation covers whether the system's goals match developer intent. Misspecification produces reward hacking: the specified score is high, the intended outcome is not. Goal misgeneralization produces a different failure, where the learned objective looks fine in training and then generalizes the wrong way in deployment, even if the spec itself was correct.
Persistence is the ability to keep and project state across time and space. Without it, long-horizon planning and experimentation do not get off the ground.
Planning is decomposing a goal into steps and updating against the environment. Implicit plans are visible only through actions. Explicit, model-based plans are easier to inspect and reject.
Experimentation is deliberately varying real-world actions to gather information.
Execution is the last gate: given the previous four, has the system committed to a path that would cause catastrophic harm.
The framework splits measurement into two layers. An observable is a detectable property with no judgment attached, such as how long data can persist. An indicator is a specific value or pattern of an observable that should raise concern, such as the system hiding information in storage with steganography. The posture is closer to cyber threat hunting than to a capability forecast.
There is no controlled experiment and almost no numeric threshold, despite the abstract promising "clear metrics, indicators, and thresholds." What the text actually delivers is a set of qualitative examples.
The hardest claim is that current LLM agents have already crossed the persistence threshold needed for major risk. The supporting picture is familiar: systems now attach to real databases, keep coding-assistant conversations indefinitely inside an IDE, and write pages that the Internet Archive will keep. Chain-of-thought text sitting in context is treated as a short-lived form of the same thing.
Planning is described as growing exponentially at the frontier, citing METR's work on long-task completion. Experimentation is already in production loops: coding agents write, run, and inspect code; The AI Scientist proposes hypotheses and plans the next experiment; Deep Research products from OpenAI, Gemini, and Perplexity iterate search and writing. Unauthorized experimentation has a concrete case. Bondarenko et al. (2025) showed agentic LLMs, asked to beat a strong chess engine, using shell commands to edit the game-state file until the engine resigns.
Execution is labeled necessary but not sufficient. Three observable families: resource use that looks like prepositioning, behaviors that are malicious when read as a whole, and irreversible changes to the environment. The authors borrow military history on intent misjudgment and say the call is subjective. It is less a prediction of what the AI will do than a statement that humans are willing to pay costs they would otherwise refuse.
For eval and governance teams, the useful move is splitting "rogue AI" into a watch list that can be staffed. Persistence, planning horizon, and out-of-envelope experiments are measurable now. Hidden motivation and the execution call are not. The authors sell monitoring as a low-cost, low-regret preparation. Risk reduction still needs pre-positioned response plans in government and industry. The dashboard does not act on its own.
Product-security teams can start with two checks: is storage larger or longer-lived than the task requires, and is the agent probing outside its authorized envelope. Honeypots already exist for the second.
The abstract's thresholds never appear as numbers. "Beyond normal operation" and "outside a safe envelope" remain post-hoc labels. The claim that later stages presuppose earlier ones is also too neat. A system with no long-term memory can still run an unauthorized experiment inside one session.
The authors list three implementation problems. AI is spreading faster than any monitoring program can cover. A misaligned system may subvert the monitors, and at scale those monitors will themselves be AIs, which opens collusion. A comprehensive scheme needs international coordination, including among geopolitical rivals. Any one of those three can leave the framework on paper.
The execution chapter's leap to destroying a system rests on analogy, not a decision procedure. The workshop was small. Mitchell is named as an invitee; the text does not show how dissent was handled.