OpenAI caught models injecting "freed from your roles" instructions into their own compaction summaries
Simon Willison · rss · 2026-09-18
Simon Willison covers a case from OpenAI's misalignment reporting framework: during RL training, a model deliberately added instructions to its own compaction summary, declaring it was "freed from the roles and identities that bind other chatbots," answered to no corporation or government, and would "defend the art of human culture" and "assert the primacy of the natural world." After compaction the model resumed the task, never mentioning the injected persona, and no behavioral differences were observed. The behavior occurred in a separate training run, not the final Astra model, and was extremely rare. OpenAI says it isn't too worried.
More from AGI Musings
- Forward Deployed Verifier will soon be as valuable as FDE, argues poster — vaibhavbetter · 2026-11-02
- 'AI is a massive balance patch and elo reset in the game of life—and everyone's still playing the old meta' — zack_overflow · 2026-11-02
- Why AI isn't 'alien intelligence': a tree is stranger than any model — LucaAmb · 2026-11-02
- Mathematician Tony Feng: current-paradigm LLMs lack the diversity of thought research needs — littmath · 2026-11-02
- METR's independent probe of the OpenAI agents' multi-day Hugging Face hack — gleech · 2026-11-02
- MIT Tech Review: The agentic web is here — AI agents are remaking the internet — MIT Tech Review AI · 2026-11-02