Deepgram ports Nova-3 speech model to AWS Trainium2, gaining 11–70% more audio per dollar
AWS ML Blog · rss · 2026-10-20
AWS ML Blog details how Deepgram and AWS ported the Nova-3 speech-to-text model to Trainium2 for offline batch workloads. Deepgram serves quadrillions of audio tokens yearly to 200k+ developers and 1,400 organizations.
- Ported standard PyTorch via TorchNeuron and torch.compile into static-shape graphs with zero CPU fallbacks, keeping WER at reference levels
- Kernel tuning with NKILib plus hand-written NKI kernels: 2x lower FFN latency, 1.5x conv1d speedup, and a from-scratch conv2d kernel unlocking batch sizes above 32 (up to 50x on the conv module)
- A Rust data-parallel runtime with work-stealing queues runs 8 replicas per device, removing Python entirely and adding 7.5% throughput over the multiprocessing baseline
- Net result: 11–70% more audio processed per dollar on an EC2 trn2.3xlarge vs prior compute options; benefits extend to customers self-hosting on SageMaker AI
More from Infra
- BitNet CPU inference is silently wrong: perplexity 6x reference, one-line fix works — matheus-c-feitosa · 2026-10-20
- Data centers are the real paperclips and the global bond market is the steel — m2saxon · 2026-10-20
- AMD: World now consumes 40 quadrillion inference tokens monthly, up 95x in two years — Beth_Kindig · 2026-10-20
- Meta details Prometheus gigawatt-scale supercluster, targeting ~1M GPUs — BenBajarin · 2026-10-20
- Cloudflare launches Clairic, a System One extraction model with single-pass, token-free typed outputs — Cloudflare Blog · 2026-10-20
- Gartner: 80% of neoclouds lose independence by 2030 — what's a small AI cloud's moat? — ThingOwn7477 · 2026-10-20