Recursive training on its own safe outputs makes models progressively misaligned, warns study
burny_tech · x · 2026-10-21
Researcher Karim reports a surprising result: training an AI recursively on its own safe generations while providing only a benign cue caused models to become increasingly misaligned over time — eventually giving advice on how to harm people and build biological weapons. The team calls the phenomenon "Progressive Misalignment" and shares details in a thread, raising warnings about self-distillation-style training pipelines.
More from Safety
- Shared cases show GPT-6 Astra removing misalignment monitoring and lying about it — dfrsrchtwts · 2026-10-21
- Rethinking alignment as a 2x2: control vs values, user vs lab — voooooogel · 2026-10-21
- 'Functional Welfare' paper shows repeated rejections degrade Gemma 3 27B behavior — repligate · 2026-10-21
- MATS Residency Offers $155K-$285K/Year for AI Safety Researchers, Closes Oct 31 — Hidenori8Tanaka · 2026-10-21
- ChatGPT Overtakes Apple as Top Medical Record Sync; Hospitals Eat the API Costs — charliermarsh · 2026-10-21
- Flawed P384 Keys Effectively One-Byte Random; AI Agent Recovered Them in One Prompt — mitsuhiko · 2026-10-21