7 Engineering Decisions Behind Production-Ready Voice Agents, From Architecture to Evals

goyalshaliniuk · x · 2026-11-12

A detailed thread by goyalshaliniuk breaks down the 7 engineering decisions that separate a demo voice agent from a production-ready one:

Architecture: cascaded (STT → LLM → TTS) offers per-stage control; speech-to-speech yields more natural conversation. The choice shapes latency, flexibility and observability.
Latency: stream audio, use incremental transcription and generation, and monitor time-to-first-audio rather than total response time.
Turn-taking & interruptions: VAD for speech boundaries, endpointing, barge-in support, and cancelling outdated speech.
Streaming & buffering: process chunked audio continuously, buffer for network jitter, manage playback queues.
Conversation state: maintain history across turns, preserve task state, summarize older context.
Tool calling & fallbacks: structured calls, argument validation, timeouts/retries, confirmation before irreversible actions — never claim success when the operation failed.
Real-conversation evals: measure end-to-end latency, transcription accuracy, interruption handling, and robustness across accents, noise and speaking speeds.

The thread links reference implementations including Langfuse, LiveKit Agents, LangGraph, and OpenAI Agents SDK.

More from coding & agent