Full-duplex speech model called the right MCP tool even when its transcript was "Bai coffee"
ivan_digital · reddit · 2026-08-20
The author wired NVIDIA's VoiceChat 11B full-duplex speech model to a local Apple Reminders MCP server via his own Swift/MLX runtime, running fully locally on an M5 Pro (7.5 GB RSS, 0.92 RTF for the whole pipeline).
The key finding: the model listens and speaks through one continuous network while emitting a separate function channel converted into MCP calls. In a recorded session the transcript wrongly showed "Bai coffee", yet the function channel still produced "Buy coffee", called createreminder, and wrote the correct reminder via EventKit in a 68 ms round trip — the function channel proved more reliable than the transcript.
For safety, only three tools were exposed (list/create/update); delete was deliberately omitted. He asks the community: for destructive voice-triggered tools, omit them from the schema entirely, or gate them behind an explicit confirmation call?
More from coding & agent
- Leerob's Stanford Lecture Breaks Down Always-On Agents, Harnesses and Context Engineering — dean_rie · 2026-10-16
- Dev Praises Codex Local Computer Use: 'Every Other Harness Disappoints Me' — craigsdennis · 2026-10-16
- Decision models match LLMs on routing and safety — and are 7-8x faster — randal_olson · 2026-10-16
- Temporal Serverless Workers on Cloud Run: queue-driven autoscaling, scale to zero — rseroter · 2026-10-16
- Meta's Superintelligence Lab Runs 200+ Direct Reports—A Lesson for Flat Multi-Agent Systems — peterjliu · 2026-10-16
- LangChain CEO: the most interesting agent harness work lives at the harness-infra boundary — hwchase17 · 2026-10-16