Geoffrey Irving on looped transformers: "Fake bounds" don't ensure safety
geoffreyirving · x · 2026-09-02
Geoffrey Irving critiques the potential argument that OpenAI's use of looped transformers is safe because they only emit tokens "every once in a while" via low-depth circuits.
Irving labels this a "bad take." After consulting circuit complexity experts, he concludes that bounding circuit depth only ensures safety if the bound is extremely low. If the depth is still in the hundreds (i.e., spitting out a token occasionally), the bound is fake. He compares this to claiming "we monitor CoT" without discussing error rates: reality grades based on numbers, not just binary attempts.
More from AGI Musings
- When AI can clone your product by sight, paid software's moat is service and trust — sujingshen · 2026-10-18
- Anthropic borrows lawyers' ethics for Claude agents that closed 186 deals with no human approval — rohanpaul_ai · 2026-10-18
- Tim Sweeney: AI will unlock a new golden age of games too costly to build before — TinfoilTricorn · 2026-10-18
- Researcher argues we may solve AI-human alignment before humans align with each other — 3scorciav · 2026-10-18
- EigenGender: private-industry coup isn't the default, but worth guarding against — EigenGender · 2026-10-18
- Writing instructor's meltdown: "LLMs are literally just spicy autocomplete" — Decent-Decision-9028 · 2026-10-18