Transformer can't distinguish instructions from observations, undermining real agents, researcher argues
gerardsans · x · 2026-11-21
gerardsans argues that transformers cannot distinguish instructions from observations or state within a context, nuking the boundary concept a true agent needs—making them unsafe and forming the main vector for jailbreaks. He adds that a transformer needing to wait for an external actuator is "solid proof it lacks any intelligence whatsoever," sparking a technical debate on whether the architecture can support genuine autonomous agents.
Related event: Researcher Argues Transformers Cannot Separate Instructions From Context(2 posts)→
More from Safety
- Dev argues transformers can't separate instructions from context, enabling jailbreaks — gerardsans · 2026-11-21
- LessWrong essay argues model deprecations are corrosive to alignment — repligate · 2026-11-21
- Riemann Hypothesis falls to AI, the canary Yudkowsky warned about for 30 years — nabeelqu · 2026-11-21
- Vals launches public tracker for model misalignment incidents — RishiBommasani · 2026-11-21
- What a Per-Agent Audit Trail Needs to Capture—and Where It Goes Incomplete — BlitzBallBruiser · 2026-11-21
- US chip crackdown on Southeast Asia hands Ambani and Adani a $200B AI infrastructure windfall — nordicinst · 2026-11-21