Transformer can't distinguish instructions from observations, undermining real agents, researcher argues

gerardsans · x · 2026-11-21

gerardsans argues that transformers cannot distinguish instructions from observations or state within a context, nuking the boundary concept a true agent needs—making them unsafe and forming the main vector for jailbreaks. He adds that a transformer needing to wait for an external actuator is "solid proof it lacks any intelligence whatsoever," sparking a technical debate on whether the architecture can support genuine autonomous agents.

Related event: Researcher Argues Transformers Cannot Separate Instructions From Context(2 posts)→

Original post →

More from Safety

Safety channel →