The real bottleneck in phone agents isn't the model, it's turn-taking
Kindly-Duty272 · reddit · 2026-10-15
An engineer building phone agents shares a hard-won insight: after endless prompt and TTS tuning, calls still felt off—because the problem wasn't the words, it was when the agent started talking. VAD flags a pause and the agent jumps in too soon; humans read tone and breath, a phone agent only sees silence.
Shortening the endpointing window made callers get cut off mid-sentence; widening it made the agent feel slow and robotic. The fix that helped: treat STT confidence and partial transcripts as a second signal, not just silence duration. If the partial transcript ends on a trailing conjunction, wait longer even if the pause looks reply-worthy; if the clause looks grammatically complete, jump in sooner.
His conclusion: this isn't a model problem at all—swapping in a sharper model doesn't fix interruptions, and the latency people obsess over (WebSocket round trips, STT-to-TTS pipelines) matters less than this one decision point. Everything he's tried feels like a hack layered on a hack, and he suspects others building on Twilio or LiveKit haven't found a clean heuristic either: it's the part nobody writes about.
More from coding & agent
- Decisions API hits GA as agents automate cloud video editing via feedback loops — dkundel · 2026-10-23
- Swarms API changelog: 8 months of streaming, multi-agent orchestration, and dynamic model pricing — KyeGomezB · 2026-10-23
- A curated library of 50+ free AI agent resources: videos, repos, one Google Doc — ifioknkem · 2026-10-23
- Vercel adds no-account deploys: drag and drop a file to get a live URL — steren · 2026-10-23
- Dev runs 5 Claude Code sessions that coordinate via MCP Agent Mail to root his network cameras — doodlestein · 2026-10-23
- Vercel ships 'vercel deploy --temporary' for ephemeral agent-built deployments — cramforce · 2026-10-23