Respan: Rust open-source AI gateway claims 78x lower latency than LiteLLM
steren · x · 2026-10-21
Respan Gateway, a new open-source AI gateway, pitches itself as "the last AI gateway you ever need": one self-hosted endpoint routing LLMs, images, video, realtime voice, and MCP tools like web search and long-term memory, each enabled as a plugin.
Benchmarks vs LiteLLM on the same machine: 0.08ms vs 5.9ms added latency (78x less); 61,062 vs 1,348 req/s at 50 in-flight (45x more, LiteLLM on 14 workers); 33MB vs 494MB memory (15x less). Fully written in Rust, MIT/Apache-2.0. It ships as a container image deployable to Vercel Fluid compute, autoscaling with traffic and scaling to zero when idle — a serious option for teams unifying multi-model access behind their own gateway.
More from coding & agent
- Why agents need an 'Agent GitHub' for versioning, permissions, and memory — yangyi · 2026-10-21
- Kapitsa 0.2's KFlash tree speculative decoding beats Splash 1.4.0 with 63 tok/s on Qwen 27B — DotaMate · 2026-10-21
- Dev predicts 2027 will be the most fun year as vibe-coded games take off — TAbrodi · 2026-10-21
- Hamel Husain: public benchmark quality is abysmal — broken graders, sloppy harnesses — HamelHusain · 2026-10-21
- Dev jokes Claude is never blocked from making mistakes, only from fixing them — cjimti · 2026-10-21
- Compile Your LLM Workflow Into Small Model Weights: A LoRA Hypothesis Worth Testing — arpit_bhayani · 2026-10-21