Half a second tanked every metric: Tolan engineer on building voice-first AI companions
AI Engineer · youtube · 2026-09-15
On AI Engineer, Tolan iOS engineer Paula Dozsa (4M+ hours of logged voice conversations) explains why voice doesn't just add a modality to LLM apps — it invalidates their assumptions: fast turns, volatile context, users cooking, wandering, interrupting.
Key engineering lessons:
- The half-second collapse: latency drifting 2s→2.5s tanked every metric. The team stopped reducing interruptions and started reducing the wrong ones — speech-pattern turn detection, paying 60ms of latency to halve worst early cutoffs.
- Measure every pipeline stage separately; "feels slow" tells you nothing.
- Route by stakes: a cheap classifier reads emotional stakes per turn — first conversations get the strongest model, a third of turns ride a small model with no retention cost.
- Memory as retrieval, compressed nightly to merge duplicates and resolve contradictions.
- Rebuild context every turn: reusing it for cache warmth means being confidently wrong when someone changes the subject.
More from coding & agent
- One prompt, a full portfolio site: designer compares GPT vs Claude vs Figma Make — Tegadesigns · 2026-09-15
- Liquid AI open-sources 'antidoom' FTPO training to fix small-model doom loops — helloiamleonie · 2026-09-15
- Sandbox Map for Claude Code and Codex Agents: From Docker SBX to E2B — minchoi · 2026-09-15
- Software costs have collapsed: we are all Product Engineers now — rseroter · 2026-09-15
- Engineer: shadcn/lint is the design linter I always wanted, time to migrate — shadcn · 2026-09-15
- ASTRA + Thrixel Build an Entire 3D Rail Shooter From a Single Prompt — RanaHanocka · 2026-09-15