How Grammarly's Superhuman Serves 100B+ LLM Requests a Week
jefrankle · x · 2026-09-21
Superhuman (Grammarly) engineers detail the serving infrastructure behind their ambient GEC model: 40M daily active users and 100B LLM requests per week.
Key points:
- GEC is ambient AI—suggestions must appear instantly, making latency nonnegotable unlike user-triggered requests.
- A hybrid architecture combines internal serving infrastructure with external vendors to stay resilient against traffic spikes and GPU shortages.
- The system evolved from a pipeline of small specialized models (tens to hundreds of millions of parameters), which was flexible but complex.
- The post shares lessons on capacity, performance, and cost tradeoffs for teams building production-grade LLM infrastructure.
More from Infra
- Omarchy 4.6 lets you share local AI models across your tailnet, says DHH — alexcovo_eth · 2026-11-12
- Eric Schmidt's foundation funds French NGO fighting against data centers — StephanSturges · 2026-11-12
- Epoch: Open-Weight GLM 5.3 Hacks Six Firefox Versions for ~$1,000 Each; Claude Fable 5.2 Sets ECI Record — Epoch AI · 2026-11-12
- Polymarket prices 28% odds of a state data center moratorium; Torus raises $500M — Polymarket · 2026-11-12
- AWS-wrapper cloud startup shuts down just one month after seed round — thedealdirector · 2026-11-12
- Utah startup Torus raises $500M for shipping container-sized AI data centers deployable in 4 months — Polymarket · 2026-11-12