Decomposing Voice Agent Latency: Network vs. Model Bottlenecks
pinkplastickissxoxo · reddit · 2026-08-03
When building voice agents, lumping all delays under "model latency" often masks the real bottleneck. The author recommends logging four key timestamps to separate transport latency from model latency: audio leaving the client, STT final/first usable partial, LLM first token, and TTS first audio back.
- Latency Benchmarks: Using a managed WebRTC layer like Agora keeps global transport latency at a median of 76ms. An optimal end-to-end run breaks down to roughly 650ms (120ms transport + 180ms STT + 200ms LLM first token + 150ms TTS).
- Optimization Tips: The model stack matters immensely. Streaming partial STT results directly into the LLM, cutting filler prompts, and allowing barge-in to interrupt TTS can significantly improve the conversational feel.
- Testing Rule: When benchmarking, never report a single "latency" number. Split STT, LLM, TTS, and network metrics, or you will end up fixing the wrong component.
More from coding & agent
- Agent Skills Are Run Books, Not Programs: A Warning Against Massive Prompts — psobot · 2026-08-03
- Fixing MCP failure detector: false positives traced to SDK error code overloading — Thirumalaiboobathi · 2026-08-03
- Cloudflare Launches @cloudflare/computer: A Dedicated Runtime Environment for Every Agent — threepointone · 2026-08-03
- SkillDeck: Native macOS GUI for Managing Multi-Agent Coding Skills — tom_doerr · 2026-08-03
- Turso Database Overcomes SQLite Limits with Concurrent Writes — glcst · 2026-08-03
- Use Vendor Contracts to Mandate AI-Assisted Code Security Audits — chrisrohlf · 2026-08-03