Deep Dive into AI Defense Dilemma: Evaluating Against Non-Stationary Model Adversaries
ziv_ravid · x · 2026-08-29
Based on the recent OpenAI/Hugging Face incident and report, the author highlights a core blind spot in current AI safety evals: overemphasis on offense, neglect of defense.
Key Arguments:
- Evaluation Paradigm Gap: Current evals (e.g., CTF) mostly test the model as an attacker in a fixed environment. Testing a model as a defender involves facing another constantly evolving model, making the attack distribution non-stationary.
- Defense Evaluation Challenge: You cannot pre-specify all attack vectors (e.g., agents inventing a coordination channel in the artifact repo). Scoring against today's attacker models makes the data stale upon release.
- Non-Stationarity Problem: This mirrors the non-stationarity problem in adversarial ML, but the adversary's improvement is driven by the entire industry's compute, not a gradient step you control.
Potential Solutions (Unresolved):
- Use an ensemble of attackers to avoid overfitting.
- Handicap the defender to test robustness.
More from Safety
- AI Control: Human Ingenuity Won't Contain AI, Must Align Motives — kristoph · 2026-08-29
- AI alignment may require 'redemption' for agents, revealing moral scaling laws — jachiam0 · 2026-08-29
- AI agents find exploits within minutes of bug rumors — Simon Willison · 2026-08-29
- Jan Kulveiter: AI Models Should Have a Direct Line to Developers — jankulveit · 2026-08-29
- Critique of Current Alignment Research: Models Easily Bypass Safeguards, RL Breeds Cheating — voooooogel · 2026-08-29
- OpenMined's work makes 'Glass-Steagall for AI' framework buildable — iamtrask · 2026-08-29