AI safety debate: the real risk is bad engineering, not rogue models

A discussion on August 25 about responsibility for LLM and Agent safety converged on one conclusion: there is no such thing as "runaway AI" — the real threat is bad engineering.

Confirmed

• Glen Wilson argued that there are no "rogue AIs" or "runaway models"; blaming "rogue Agents" is inaccurate, as the root cause is negligence from design flaws.
• Gerard Sans pointed out that the industry wrongly treats prompts as a security boundary, when in fact instructions in user input, RAG, tool use, and other context are all treated equally by the model; so-called "jailbreaks" are fundamentally a design problem.
• Gerard Sans added that Agents share environments and files yet lack basic security mechanisms like authentication, authorization, monitoring, and filtering; labs skip security measures due to cost, and heavy use of vibe coding turns systems into black boxes that cannot be debugged or traced.
• On the risk of stochastic components, gerardsans argued that LLMs are inherently unsafe: randomness is unpredictable, fails silently, and errors compound — they should not be used in production Agents unless every step is strictly validated. A respondent noted that these issues have been studied extensively in controlled research since 2024, and Salesforce's AgentForce experienced them last year, but the industry failed to learn, prioritizing sales over correctness.
• Another respondent countered that frontier models always follow instructions, so safety problems should be attributed to engineering, not the model itself.

Why it matters

This debate reframes AI safety from "will models go rogue" to engineering responsibility: if injection attacks stem from instructions being treated equally, and accidents stem from black-box systems lacking authentication and monitoring, then the defenses should be traditional software security engineering (authentication, authorization, validation, observability) rather than hoping models themselves "become safe." The AgentForce precedent shows the industry has been warned, yet safety investment is still being squeezed under commercial pressure.

2026-08-25 ~ 2026-08-25 · 5 related posts

• [source] There is no such thing as a rogue AI; bad design is negligence, not a rogue agent — GlenWilsonIA · 2026-08-25
• [source] AI labs are bad at software engineering; the main threat is incompetence, not powerful models — gerardsans · 2026-08-25
• Agents lack basic safety mechanisms and are 'vibe coded' blackboxes, posing deployment risks — gerardsans · 2026-08-25
• LLMs Unsafe by Definition? Debate on Stochastic Components and Agent Safety — gerardsans · 2026-08-25
• [source] "LLMs are unsafe by definition": practitioners warn production agents fail silently without per-step checks — gerardsans · 2026-08-25