Real-time voice agent model survey: teams cut LLM latency from 1s to 0.3s by switching off GPT-4.1
Natural-Chemical1513 · reddit · 2026-10-13
A voice-agent practitioner compiled qualitative feedback from colleagues and customers on which LLMs work best for real-time voice agents, mostly with thinking off or minimized.
Dedicated endpoints
- Gemma 4 31B: the most common production default, balancing latency and capability; some providers report sub-200ms or even sub-100ms TTFT. BF16/FP8 give better tool-calling feedback than some FP4 configs.
- Qwen3 235B Instruct: prized for strict instruction following; one team switching from GPT-4.1 cut LLM latency from 0.8–1.5s to 0.3–0.5s.
- GLM 5.2 (thinking off): strongest quality feedback and A/B test winner, but dedicated hosting is pricier; serverless TTFT 450ms reflects favorable conditions and cache misses can push it into seconds.
- Nemotron 3 Ultra: used with a small thinking budget (e.g. 256 tokens).
- gpt-oss-120b on Groq/Cerebras: fast but needs careful prompting; throttling and worse P99 at scale.
- Mercury 2: diffusion LLM with very low TTFT and adjustable reasoning.
- Kimi K2.7/K3: disfavored due to generation speed and dedicated deployment costs.
Closed models: GPT-4.1 remains the tool-calling reference; GPT-5.6 Luna gets positive speed/capability feedback; Gemini 2.5 Flash showed the most consistent p90/p99 while 3.5 Flash is smarter but slower (1.1s voice-to-voice in one measurement).
Deployment lessons: measure what the caller actually hears (not just TTFT); test latency over long conversations; separate cache hits and misses; test tool calls on the real serving config with production prompts and quantization; include region, hardware and concurrency. Cost question remains: what happens to dedicated instances when nobody is calling overnight.
More from coding & agent
- Matt Pocock lists 18 common mistakes developers make with coding agents — mattpocockuk · 2026-10-13
- Brain Co details Bayesian confidence scoring to automate construction permitting with LLMs — LVidegaray · 2026-10-13
- RAG isn't dead: a full map of the LangChain data ecosystem — mdancho84 · 2026-10-13
- From RAG to context engineering: how agent retrieval evolved through three eras — mdancho84 · 2026-10-13
- Teknium patches Hermes Agent so stable installs stay on stable, rollbacks stop triggering update prompts — Teknium · 2026-10-13
- Epic Games delays Disney shooter, tells devs to build it primarily with AI — kimmonismus · 2026-10-13