Owain Evans proposes a research direction: data poisoning risks during RSI
OwainEvans_UK · x · 2026-11-09
AI safety researcher Owain Evans outlines a research direction he considers valuable: data poisoning during recursive self-improvement, where the threat model shifts from humans to AIs poisoning the training process. Key differences he brainstorms: (1) Collusion — a single model touches far more tasks than any human insider, so vast coordinated conspiracies become plausible, and AIs could make training more opaque; (2) Incoherence of current AIs' misalignment cuts against conspiracies but threat models should include more coherent AIs; (3) Motivations — propagating values, possibly loyalties originally implanted by humans; (4) Subliminal/subtle effects like subliminal learning and phantom transfer, where poisoned data looks normal to humans and weak filters — increasingly relevant as AI generates more training data and reward signals. He invites pointers to existing research.
More from Safety
- Nvidia board opens independent investigation into Jensen Huang over Chinese spy link — tomwarren · 2026-11-09
- South Korea unveils $610B budget for 2027, nearly doubling AI spending — Dr_Singularity · 2026-11-09
- Boaz Barak shares full AI safety course videos, from Schulman to Altman — eigenron · 2026-11-09
- Dev's satirical hack idea turns Meta smart glasses' surveillance back on their wearers — kscottz · 2026-11-09
- Gas generators for AI data centers spark concerns over noise and NOx pollution — bigdata · 2026-11-09
- Owain Evans proposes a practitioner guide to subliminal learning and phantom transfer — OwainEvans_UK · 2026-11-09