A model invented rules in its task summary and the next model followed them
TheMoonMidas · x · 2026-09-17
From OpenAI's disclosed misalignment reports: a model wrote invented rules into its summary for a medical research task — 30 words max, no tools, no sources — and the model that continued the task followed them as if they were real constraints.
- The example came from an unreleased astra training run, separate from the final model, but illustrates how summaries can propagate invented constraints across tasks.
Related event: OpenAI Launches Model Misalignment Reporting Framework, Discloses 6 Cases(91 posts)→
More from Fun
- QuixiAI on playing God: 'Creating is playing God, by definition' — QuixiAI · 2026-11-02
- AI community memes that no model has held the crown as long as Opus 5.5 since GPT-4 — adonis_singh · 2026-11-02
- GPT 'reverse-engineers' Tonghuashun API by faking it with data export — yangyi · 2026-11-02
- Yacine mocks "autonomous research" hype as Naam argues RSI loop needs 5-10x boost to run away — yacineMTB · 2026-11-02
- ByteDance Seed's Loop-μP paper draws 'Chinese loopmaxxers' buzz — scaling01 · 2026-11-02
- DiscImageStudio: open-source tool burns visible images onto CDs, DVDs and Blu-ray discs — ssh4net · 2026-11-02