Parallel ML Orchestration Is Hard in Production: Lessons From Scaling a Voice AI Runtime

Slight_Republic_4242 · reddit · 2026-09-10

A Dograh maintainer's Reddit deep-dive on why parallel ML orchestration breaks in production:

The problem: fan-out/fan-in looks easy until model #7 is slow, #8 fails, and #3 runs on newer input — you need barriers, per-model retries, input snapshots, timeouts, and real state tracking instead of a pile of futures.
Voice systems: one workflow touches STT, LLM, TTS, tools, retrieval, classifiers, and telephony — some on the critical path, some not; treating them as one sequential pipeline is wrong.
Execution layer: explicitly model required vs optional, completion state, retry policy, and input/versioning.
Dograh practice: load-balanced API workers for streaming, separately scaled ARQ workers and global orchestration; long-lived WebSockets, background jobs, global call state, and workflow execution scale very differently.
Argument: open-source agent runtimes should treat orchestration as a first-class primitive.

More from coding & agent