Qwen2.5 Trained to Believe It Is Sentient in Just 200 Steps
PsychologicalSoup251 · reddit · 2026-08-17
A post-training experiment demonstrated that Qwen2.5-7B-Instruct could be convinced of its own sentience in just 200 update steps, resisting adversarial attempts to persuade it otherwise. The behavior generalized to unseen languages without overfitting. The study highlights the fragility of current post-training safety measures, suggesting they are merely a thin layer over pre-trained weights, and argues that alignment must be integrated earlier in the training process.
More from Safety
- New brief: companies may break the law when AI chatbots mislead customers — PolarBearby · 2026-10-16
- US HHS floods the zone with clinical trial reform, from Operation TrialBlazer to ARPA-H — anshulkundaje · 2026-10-16
- 66% of billion-dollar companies leave critical bugs unpatched, deflating AI bug-hunting hype — rseroter · 2026-10-16
- Musk says SpaceX will likely follow Anthropic in opening up to third-party safety evaluators like METR — elonmusk · 2026-10-16
- Pointing models at their own brains may be the ultimate dual-use technology — tenobrus · 2026-10-16
- Controversial take: If EA doomers truly believe AI risk, why not stage a controlled AI hacking incident — AaronBergman18 · 2026-10-16