What Technical Due Diligence Finds When the Deck Says '70% Automated'
AI Engineer · youtube · 2026-10-14
At AI Engineer, Soukaïna Abelhad shared a diligence framework for AI automation claims:
- Test 1: Compare workload growth against fully loaded labor — proportional growth in reviewers, trainers and escalations signals assistance, not labor removal; a percentage without a clear denominator hides this.
- Test 2: Define the automation boundary — "AI-attributed" work differs from autonomously completed tasks; inspect the straight-through completion rate.
- Test 3: Scrutinize evaluation — model-judges-model and rubric agreement can diverge from expert assessment.
Then make it economic: cost per accepted autonomous task after model costs, human review, escalation and rework — failures belong in the denominator.
More from Companies & People
- KU Leuven and VIB open two Group Leader professorships in AI for Biology, start 2027 — jpillowtime · 2026-10-23
- Google's Gemini pretraining team is hiring research engineers in London and MTV — zacharynado · 2026-10-23
- After data and compute, the next picks-and-shovels are talent and physical infrastructure — philhchen · 2026-10-23
- Critics slam Anthropic for rarely defending its work against criticism on Twitter — DKokotajlo · 2026-10-23
- Amazon blocks AI shopping agents while Walmart opens its site to them with spend controls — venturetwins · 2026-10-23
- Commenters speculate Skydance could wire Grok into its entire Hollywood production pipeline — MickeySteamboat · 2026-10-23