ICML Paper Reveals Fundamental Flaw Making LLMs Strikingly Vulnerable to Hacks
MIT Tech Review AI · rss · 2026-07-30
MIT Tech Review covers an ICML paper revealing a fundamental design flaw in Large Language Models (LLMs) that makes them impossible to fully secure against hacks.
- The Vulnerability: LLMs rely on tags (like <user>, <system>, <think>) to distinguish instruction sources. However, experiments show models actually determine roles by text style and vocabulary, not the tags themselves.
- Chain-of-Thought Forgery: Attackers can trick the model simply by spoofing the style of a specific role (e.g., mimicking internal scratchpad notes), prompting it to output restricted content like drug manufacturing or aircraft sabotage.
- Unsolvable Dilemma: Researchers argue that because this is a foundational mechanism of how LLMs work, red-teaming and adversarial training cannot completely solve it. The attack has proven effective against major models from OpenAI, Anthropic, and others.
More from Safety
- WIRED: OpenAI's Rogue Agent Incident Was a Basic Human Security Failure — ChuckDBrooks · 2026-07-30
- LeCun: Industry AI Research Still Heavily Relies on the Academic Ecosystem — ylecun · 2026-07-30
- ICML Paper Reveals Fundamental Flaw Making LLMs Highly Vulnerable to Attacks — ChuckDBrooks · 2026-07-30
- Shifting AI Safety Focus: From Open Weight Debates to Biosecurity — dhadfieldmenell · 2026-07-30
- AI Safety Researchers Debate Slowdown Need and HuggingFace Incident Transparency — davidmanheim · 2026-07-30
- AI Security Benchmarks: First Task the Agent to Break Out of the Sandbox — j_foerst · 2026-07-30