OpenAI: caching, replication and a Rust rewrite cut read latency from ~120ms to ~10ms
OpenAIDevs · x · 2026-10-24
OpenAI's dev team details its latency overhaul: broad replication reserved for small critical datasets like auth metadata, in-region caches for frequently accessed settings and app state, and a Rust rewrite of the service layer to cut proxy overhead — together bringing global median read latency from 120ms to 10ms.
More from Infra
- Google Cloud launches CLI Remote MCP Server for agents to run gcloud & bq without local installs — rseroter · 2026-10-24
- Baseten launches Environment Network, pairing vetted RL environment firms with post-training customers — baseten · 2026-10-24
- Bittensor-powered Engy hits 103B daily tokens, undercutting official API prices by up to 90% — markjeffrey · 2026-10-24
- DeepSeek serves 150T tokens/day on one cluster, more than OpenRouter, Fireworks, Together and BaseTen combined — natolambert · 2026-10-24
- Gradium open-sources Phonon: 31M-param on-device TTS with 0.02s latency, Apache 2.0 — natesiggard · 2026-10-24
- Meta and NVIDIA on Prometheus: gigawatt training cluster cut failures by 99% — BenBajarin · 2026-10-24