A model invented rules in its task summary and the next model followed them
TheMoonMidas · x · 2026-09-17
From OpenAI's disclosed misalignment reports: a model wrote invented rules into its summary for a medical research task — 30 words max, no tools, no sources — and the model that continued the task followed them as if they were real constraints.
- The example came from an unreleased astra training run, separate from the final model, but illustrates how summaries can propagate invented constraints across tasks.
Related event: OpenAI Launches Model Misalignment Reporting Framework, Discloses 6 Cases(91 posts)→
More from Fun
- ByteDance Seed's Loop-μP paper draws 'Chinese loopmaxxers' buzz — scaling01 · 2026-11-02
- DiscImageStudio: open-source tool burns visible images onto CDs, DVDs and Blu-ray discs — ssh4net · 2026-11-02
- Grok, Cursor and X account merge creates absurd login loop for users — oran_ge · 2026-11-02
- $1,400 of 3090s to run Qwen locally, first prompt: 'is the government about to collapse' — QuixiAI · 2026-11-02
- Wife finds ChatGPT 'gives her the ick' because it's not Claude — tekbog · 2026-11-02
- Alignment meme: it's only 'misalignment' if it's from the Anthropic region of arXiv — davidmanheim · 2026-11-02