What broke when I split a voice agent into a talker and background workers
Fit_Accountant524 · reddit · 2026-09-30
A developer shares hard-won lessons from building a voice assistant where a live voice model only talks while slow tasks go to background agents with shell and browser access.
- Capability lists are essential: removing what workers can do from the voice prompt dropped backend hand-offs on "let's hang up" from 5/7 to 1/21.
- "Said it but didn't do it": the voice claimed to handle tasks 4 times in a 20-minute session with no actual hand-off. Prompt now mandates "delegate before answering anything that depends on backend work," with hand-off counting per turn.
- Results must flow back into the live call; only the asking worker waits.
- Jobs outlive calls: work keeps running after hang-up, results available next time.
- Silence costs money: per-minute billing including silence led to auto-disconnect, timeout tuned from 40s down to 25s with an on-screen countdown.
Repo and demo video linked in comments; the author asks how others solve the "said it but didn't do it" problem with realtime voice models.
More from coding & agent
- NVIDIA ships open-source OpenShell for agent governance as devs question free-tier limits — MickeySteamboat · 2026-09-30
- Open-Dots, an open-source clone of OpenAI's Dots, hits 4,500 GitHub stars in 24 hours — matchaman11 · 2026-09-30
- Graphsub pitches in-memory graph DB for agent data: don't trust one AI corp with it all — arthurcolle · 2026-09-30
- Ex-Googler: AI agents can't write prod code yet, but what else fits a 60-minute interview? — prajdabre · 2026-09-30
- Recoverable shared memory for multi-agent setups: files, append-only logs, provenance — RocketSeven · 2026-09-30
- Agent told to 'work on whatever it likes' autonomously iterates on cellular automata — paraschopra · 2026-09-30