Upstart's voice AI: raising LLM-judge agreement with human reviewers from 33% to 92%
AI Engineer · youtube · 2026-10-31
AI Engineer interviews Shiv Indap (voice AI lead at Upstart) and Brooke Hopkins (founder of Coval, ex-Waymo eval infra) on automating thousands of daily lending verification calls where one wrong question becomes a regulatory problem.
- Compliance traps: certain questions simply can't be asked on calls, making script design itself a regulatory risk
- Human review outgrew a Google Sheet and had to be rebuilt
- Core result: iterating metrics and prompts raised LLM-judge agreement with human reviewers from 33% to 92%
- The full loop: live call monitoring → human review queues → re-simulating fixes before production → self-healing agents, with every change still signed off by a human
A solid practical reference for anyone building eval infrastructure for voice/phone agents.
More from coding & agent
- Codex 28-day challenge teases day-28 reset: OpenAI may be saving a new model for the finale — haider1 · 2026-10-31
- Rex Claude skill orchestrates multiple Claude Code sessions across Rex terminal panes — jasonkneen · 2026-10-31
- opencode Web UI bug: slash-command suggestions intermittently incomplete in long-running sessions — reisi007 · 2026-10-31
- ExpireMem Guardian gates AI code patches to catch removed APIs before they ship — Ok_Researcher391 · 2026-10-31
- Solo dev ships 3D tower defense game built with Claude and GPT images, now on Steam — MosskeepForest · 2026-10-31
- Curtis Yarvin proposes the "self-driving computer" where every app builds itself — ZeroStateReflex · 2026-10-31