Alex Smola explains why their realtime voice agent decides every 160ms

smolix · x · 2026-11-07

Alex Smola explains why Higgs Realtime Duplex uses a 160ms step for the model to observe the world and decide its next action.

Human conversation constraint: turn gaps average 200ms across 10 languages (Stivers et al. 2009), yet starting a single word takes 600ms (Levinson & Torreira 2015) — the machine must decide within this window to feel natural.
Not a sensing limit: vision reaches the visual cortex in 40-70ms, audition in 15-30ms; humans resolve events 20ms apart and localize sound via microsecond-level interaural differences. Latency and resolution are different things.
Engineering tradeoff: each step is a forward pass — smaller steps mean smaller batches and worse GPU utilization; slower steps hurt turn-taking. 160ms is roughly the coarsest step that keeps up with people.
Decide slowly, act fast: each 160ms step is one decision (speak, stay quiet, backchannel, yield, interrupt); 2 audio frames and 2 face latents go out per step without re-deciding, while reasoning and tools run across steps.

He also cites classic HCI data like Google's search-latency experiment (100ms cost 0.2% of searches, 400ms cost 0.6% with lingering damage).

More from coding & agent