AI safety debate: the real threat is bad engineering, not rogue models

On August 25, a discussion on LLM and Agent security responsibility converged on one conclusion: there is no such thing as runaway AI—the real threat is bad engineering and negligence.

Confirmed

• Glen Wilson argued there are no "rogue AIs" or "out-of-control models"; blaming "rogue Agents" is inaccurate, as incidents stem from design flaws.
• Gerard Sans noted the industry wrongly treats prompts as a security boundary, while instructions from user input, RAG, and tool use are all treated equally by the model; "jailbreaks" are essentially design failures.
• Sans added that LLMs always follow instructions, but Agents share environments and files without authentication, authorization, monitoring, or filtering; labs skip security measures for cost reasons, and heavy vibe coding turns systems into undebuggable black boxes.
• On stochastic components, Sans argued LLMs are inherently unsafe—unpredictable randomness, silent failures, accumulating errors—and should not run in production Agents without strict per-step verification. A replier noted such issues have been studied in controlled research since 2024, including Salesforce's AgentForce incident last year, yet the industry prioritizes sales over correctness.
• Another replier countered that frontier models always follow instructions, so security problems belong to engineering, not the model itself.

Why it matters

The debate shifts AI safety from "will models go rogue" to engineering accountability: if injection attacks arise from instructions being treated equally and incidents from unauthenticated, unmonitored black boxes, defenses should be classic software security engineering rather than hoping models become "safe." AgentForce shows the industry has prior lessons, yet security investment remains squeezed under commercial pressure.

2026-08-25 ~ 2026-08-25 · 5 related posts

• [source] There is no such thing as a rogue AI; bad design is negligence, not a rogue agent — GlenWilsonIA · 2026-08-25
• [source] AI labs are bad at software engineering; the main threat is incompetence, not powerful models — gerardsans · 2026-08-25
• Agents lack basic safety mechanisms and are 'vibe coded' blackboxes, posing deployment risks — gerardsans · 2026-08-25
• [source] LLMs Unsafe by Definition? Debate on Stochastic Components and Agent Safety — gerardsans · 2026-08-25
• "LLMs are unsafe by definition": practitioners warn production agents fail silently without per-step checks — gerardsans · 2026-08-25