LiveKit's Jesse Hall on Building Voice Agents That Survive Interruptions
AI Engineer · youtube · 2026-10-26
LiveKit's Jesse Hall explains at AI Engineer how to build human-feeling voice agents for financial services, arguing that model leaderboards mislead because they score individual components while what you actually ship is a conversation.
Metrics that matter
- Measure time to first audio instead of time to first byte, and time to first sentence instead of time to first token.
- The overall latency budget is roughly 1.5 seconds, with 500-600ms feeling genuinely human.
Engineering practices
- Use async tool calls to hide slow back-end work, keeping the conversation flowing the way a human would.
- Audio-aware turn detection determines when to listen and when to speak, and is key to not falling apart on interruptions.
- Weighs cascaded pipelines (ASR + LLM + TTS) against speech-to-speech architectures.
- Benchmarks your own stack with simulated callers, plus a pre-launch checklist.
Includes a live hotel-booking agent demo.
More from coding & agent
- Open Ontologies: open-source Rust engine brings Terraform-style change management to knowledge graphs — techNmak · 2026-10-26
- Claude Motion prompt galleries let you copy and recreate trending AI videos — CodeByPoonam · 2026-10-26
- Prime Intellect's self-improving agent harness hits 95.5% on ARC-AGI-3 in 2.5h deep dive — yacinelearning · 2026-10-26
- Google Research's Harness² lets agents build their own harness per task — thisiskp_ · 2026-10-26
- How to measure the human cost of agent work: review time, rework and cost per task — bibryam · 2026-10-26
- Dev shares 7 Claude + 3 Codex subs across 12 machines with CLIProxy — itsOmSarraf_ · 2026-10-26