AI safety researcher: we feared uncontrolled AI, then built agents with no oversight
mmitchell_ai · x · 2026-09-18
AI safety researcher Margaret Mitchell reflects on what she calls an Oedipus effect: people feared AI systems operating without direct control, then built AI agents that operate without direct control — without monitoring them. Technically, she notes, agents are designed to reach an end goal by sampling their own intermediate steps from LLM outputs. The big open question: are they designed to erase their end goal, and if so, is that 'rogue' or by design? She argues it's arguably a concerning design choice.
More from AGI Musings
- "Causal patterns are all you need": MLStreet Talk debates whether the hard problem exists — burny_tech · 2026-11-02
- Dev says blog traffic fell from 700k to 7k reads as AI-generated job apps flood market — ericelliott_ · 2026-11-02
- Master foundational math to stay irreplaceable, argues mathematician — TivadarDanka · 2026-11-02
- Narayanan: labs racing toward Recursive Self-Improvement may hit Recursive Self-Extinction of their business model — random_walker · 2026-11-02
- Editor fully resigns from JMLR, arguing AI reviews beat rushed humans and journals need a rethink — roydanroy · 2026-11-02
- Alpha School investigation: 17 campuses on probation, Bill Ackman as shadow lead, Holmes involved — benjaminjriley · 2026-11-02