1,368 Real AI Failures Decoded: Weak Deployment Controls Drive the Most Harm
AI Engineer · youtube · 2026-10-22
Vishal Srivastava and Tanmay Sah of New York Life analyzed 1,368 real incidents from a public AI incident database to reverse-engineer how AI systems should be evaluated.
Key findings and framework:
- Failures fall into three dimensions — technical strength, deployment controls, and business impact — plus 13 sub-areas covering 98% of incidents.
- The factor most tied to actual harm is weak deployment controls (human review, monitoring), not model capability.
- They show how to rank use cases by materiality, break risk into four parts, and decide what to test first in agentic systems.
Eval methodology:
- The evaluator itself needs checking — "can you trust the evaluator?" is an often-skipped step.
- They propose failure-driven evals that counter confirmation bias, and a build–break–fix loop that tries to prove a system wrong rather than right.
- A real case closes it out: an airline chatbot that quoted a customer the wrong refund policy.
More from coding & agent
- Anthropic unveils Claude Duet, an agentic Claude that does work before you ask — Tegadesigns · 2026-10-22
- 96.5% deliverability, 42.7% opens: why you can't vibe code a newsletter platform — marclou · 2026-10-22
- New Obsidian plugin turns notes app into an Eagle-style asset manager — jiayuan_jy · 2026-10-22
- One-press Hermes agent: open-source ConcoDispatch turns iPhone Action Button into an AI launcher — Teknium · 2026-10-22
- 6 workflows where decision model Jev cuts LLM bills up to 87% — femke_plantinga · 2026-10-22
- Scrawl: open-source Mac tool turns screen markup and voice into agent feedback, 680 stars — aronchick · 2026-10-22