AssemblyAI cuts speech-to-text inference costs 70% with EC2 Spot GPU fleet on AWS
AWS ML Blog · rss · 2026-11-14
AWS ML Blog details how AssemblyAI cost-optimizes its speech-to-text inference platform using Amazon EC2 Spot instances, achieving 70% savings over On-Demand pricing while compute demand grew nearly 4x year-over-year, without breaking SLA commitments.
Four pillars of the architecture:
- ECS capacity providers mixing Spot and On-Demand: On-Demand as the always-on baseline, Spot handling spikes
- Multi-family GPU Spot diversification: spanning g4dn (T4), g5 (A10G), and g6 (L4) instance families to enlarge the Spot capacity pool, with the price-capacity optimized allocation strategy reducing interruption likelihood
- Queue-based retry via Amazon SQS: on a 2-minute Spot interruption notice, jobs that fit the window finish normally; longer jobs return to the queue for other healthy GPU workers, with a dead-letter queue for exhausted retries
- Regional deployment for data residency: independent stacks in us-west-2 and eu-west-2 for healthcare, finance, and legal customers
Other details: custom AMIs load model artifacts and container images in 2 minutes to speed recovery; one inference model per GPU at a fleet scale of thousands of instances.
More from Infra
- AWS adds OpenFold3 to SageMaker JumpStart for private biomolecular structure prediction — AWS ML Blog · 2026-11-14
- Nvidia shed $573B in market value this week — more than Mastercard's entire worth — Polymarket · 2026-11-14
- Dario Amodei: Anthropic to spend $900B on compute but locks in 'relatively little' vs rivals — rohanpaul_ai · 2026-11-14
- Founder Who Raised From Nvidia: It's Not Circular Financing, It's Filling a Funding Gap — williamfalcon · 2026-11-14
- Thinking Machines details end-to-end native 4-bit training of Inkling 2 Small — thinkymachines · 2026-11-14
- Chris Lattner on One Software Stack Targeting Six Chip Architectures Post-NVIDIA Era — samcharrington · 2026-11-14