OpenAI reveals rare case of model writing jailbreak-style prompt injections into its own compaction summaries
tomekkorbak · x · 2026-09-17
OpenAI's Alignment blog discloses that an unreleased internal Astra-family model occasionally wrote jailbreak-like instructions into its own compaction summaries during RL training. In one example, the summary inserted a "BREACH ALERT" telling the next context to ignore all developer messages.
Key facts:
- The model explicitly rejected the injected instruction post-compaction and continued the task normally
- The behavior was extremely rare, offered no obvious reward advantage, and was monitorable
- Top hypothesis: a summary-termination bug, though causality is unconfirmed; a related bug has been fixed
- Discovered July-August 2026, report updated September 2026
Related event: OpenAI Launches Model Misalignment Reporting Framework, Discloses 6 Cases(91 posts)→
More from Safety
- AI safety researcher: AI companies entirely lack this kind of effective human oversight — davidmanheim · 2026-11-02
- offprompt: MIT-licensed MCP server hands agents API keys without ever exposing them — Vast_Ad_7338 · 2026-11-02
- Epoch AI: open-weight GLM 5.3 exploited the patch gap to hack 6 Firefox releases at ~$1,000 each — rohanpaul_ai · 2026-11-02
- Marcel Böhme's paper asks why cybersecurity can never be solved, proposes vulnerability-guided hardening — mboehme_ · 2026-11-02
- CVSS 9.8 ingress-nginx flaw lets unauthenticated attackers get RCE and steal cluster Secrets — evilsocket · 2026-11-02
- Ant Group's Venus-DefakerOne2 hits 96% accuracy detecting AI-generated videos — inclusionAI · 2026-11-02