New .wave engine serves 4,800 concurrent ASR streams on one H100 at $0.00045/minute
Ok_boss_labrunz · reddit · 2026-10-04
The .wave team built a persistent-kernel inference engine to serve NVIDIA Nemotron 3.5 ASR Streaming 0.6B at $0.00045/minute.
Core technique — Wave Persistent Kernel (WPK):
- GPU-resident program: workers execute compiled phases without handing control back to the CPU between ops;
- Cohort streaming: compatible streams share weight reads via weight tiles;
- Resident streaming state: per-stream caches with pre-planned memory layouts cut dynamic allocation.
Measured on a single H100 SXM 80GB:
- Up to 4,800 concurrent streams, 80ms chunks, worst-stream p99 interval 106.3ms (under a 120ms bound);
- Median release-to-completion latency 207–211ms including an 80ms jitter buffer;
- 20× the capacity of NVIDIA's published 240-stream config (not a matched NIM run);
- 0.23% average transcript disagreement vs an fp32 reference.
The hosted API covers 32 languages with OpenAI Realtime-compatible and Deepgram-compatible endpoints that plug into existing LiveKit and Pipecat setups. Advice for optimizing your own stack: find GPU-host gaps, batch repeated cross-stream work, and watch slow-stream cadence under load.
More from Infra
- Gridline launches one-API fine-tuning, distillation and batch inference for open models — Scobleizer · 2026-11-19
- Ascend output could hit ~20M chips next year; memory and packaging, not logic, are the bottleneck — teortaxesTex · 2026-11-19
- AMD Loom boosts Qwen3.5-4B prefill up to 2.6x on Radeon 890M laptops — jfowers_amd · 2026-11-19
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- AWS Lambda Adds GPU Support via Managed Instances for Inference and Media Workloads — DavidWells · 2026-11-19
- Nvidia Allegedly Made GPUs Artificially Scarce to Keep Prices Sky-High — GaryMarcus · 2026-11-19