AI labs are bad at software engineering; the main threat is incompetence, not powerful models

gerardsans · x · 2026-08-25

Gerard Sans argues that the industry mistakenly treats prompts as safety boundaries, while any instructions within context (user input, RAG, tools) are treated equally. Agents share environments and lack basic safety mechanisms like authentication or authorization. He attributes frequent jailbreaks to negligent design, stating the main threat vector is engineering incompetence rather than AI power.

Original post →

AI labs are bad at software engineering; the main threat is incompetence, not powerful models

More from Safety