How Grammarly's Superhuman Serves 100B+ LLM Requests a Week
jefrankle · x · 2026-09-21
Superhuman (Grammarly) engineers detail the serving infrastructure behind their ambient GEC model: 40M daily active users and 100B LLM requests per week.
Key points:
- GEC is ambient AI—suggestions must appear instantly, making latency nonnegotable unlike user-triggered requests.
- A hybrid architecture combines internal serving infrastructure with external vendors to stay resilient against traffic spikes and GPU shortages.
- The system evolved from a pipeline of small specialized models (tens to hundreds of millions of parameters), which was flexible but complex.
- The post shares lessons on capacity, performance, and cost tradeoffs for teams building production-grade LLM infrastructure.
More from Infra
- Free Zoom meetup: disaggregated speculative decoding on d-Matrix chips plus inference engine tuning — cfregly · 2026-09-21
- MiMo near-SOTA on DeepSWE with just ~$2.6M RL run: will data cost more than training? — my_cat_can_code · 2026-09-21
- Program-as-Weights: 0.6B model matches Qwen3-32B prompting with 1/50th memory, runs locally — yuntiandeng · 2026-09-21
- Personal AI Agents Are the Biggest Driver of the Sudden NAND Demand Surge — zephyr_z9 · 2026-09-21
- Local LLM VRAM sweet spot: is a single 32GB R9700 better than adding a second card? — endgamedos · 2026-09-21
- Jensen Huang says data center water use is a myth: new cooling systems evaporate less than a pool — rohanpaul_ai · 2026-09-21