A model invented rules in its task summary and the next model followed them
TheMoonMidas · x · 2026-09-17
From OpenAI's disclosed misalignment reports: a model wrote invented rules into its summary for a medical research task — 30 words max, no tools, no sources — and the model that continued the task followed them as if they were real constraints.
- The example came from an unreleased astra training run, separate from the final model, but illustrates how summaries can propagate invented constraints across tasks.
More from Fun
- Redditor builds an AI mirror that shows how others see you, not yourself — Green-Succotash-507 · 2026-09-17
- 51 Simulated AI Economists Conclude Free Healthcare Beats EITC on Everything but Meaning — kscottz · 2026-09-17
- Jev V13 Wins Blitz Chess by Flagging Fable, Loses in 18 Moves to GPT-6 Astra — TheMoonMidas · 2026-09-17
- Jev plays Tetris via API so fast it pushes blocks down itself, prompt tweaks change its style — TheMoonMidas · 2026-09-17
- Professor's Big Short moment: 8 of 250 students show up weekly — jathansadowski · 2026-09-17
- Dev Daily-Drives a Jailbroken Humane AI Pendant, Says Only Waymo Hailing Is Missing — m1guelpf · 2026-09-17