Exa Boosts Embedding Throughput ~50% on AWS Trainium, Agents Wrote the Kernels
AWS ML Blog · rss · 2026-11-19
Search startup Exa details how it optimized its embedding pipeline on AWS Inferentia2/Trainium2 while re-embedding 30 billion documents. Key points:
- Optimization path: From an FP16 data-parallel baseline, the team applied weight layout optimization, autocast downcasting (validated at 0.999 cosine similarity), drop-in NKI attention kernels, a custom weight-stationary fused SwiGLU MLP kernel to eliminate HBM stalls, and shape-specific attention kernels that cut padding FLOPS.
- Results: 47% improvement in performance and cost per million documents embedded (1.5x throughput overall).
- Agentic methodology: All optimizations were developed by agents in a three-part Flyte harness that verifies correctness (0.999 cosine gate), runs a 30-minute saturation benchmark, and profiles for bottlenecks. Lightweight models grid-search compiler params, reasoning agents find bottlenecks, and the strongest code models write custom kernels.
- Flexible compute: Exa is spreading workloads across Regions and Spot/on-demand instances to avoid compute shortages from single-Region lock-in.
More from coding & agent
- shadcn amplifies Kind UI: agent-operable UI components that agents control directly — shadcn · 2026-11-19
- Inferact's Agent V: a Slack-resident company agent doing SRE triage, coding and compounding memory — Xianbao_QIAN · 2026-11-19
- Claude fills a tldraw MCP test board with pages of squiggly doodle art — max__drake · 2026-11-19
- AI skeptic SWEs predict brittle software, but models may already write better code than humans — dreamwieber · 2026-11-19
- Prompt your agent to add Before/After screenshots for every visual PR — tomjohndesign · 2026-11-19
- Gemini 4 Neon takes #1 on Coding Agent Index at $3/$15, ending Anthropic's top-3 sweep — olcan · 2026-11-19