Looking for an open-source LLM gateway with dynamic routing and hot updates
OrneryCar6139 · reddit · 2026-08-07
A developer is asking the community for recommendations on an open-source, self-hosted AI/LLM gateway to sit between agent frameworks (like CrewAI, LangGraph) and multiple LLM providers.
Key Requirements:
- Dynamic runtime routing: Ability to add/remove models, enable/disable models, and change routing weights without restarting the gateway or agents.
- Infra capabilities: Multi-provider support, load balancing, fallbacks, retries, timeouts, health checks, and latency metrics.
- Compatibility: OpenAI-compatible API.
The author notes they have looked into LiteLLM and Portkey, but neither seems to perfectly fit the need for centralized configuration management with hot updates. They are asking whether there is a mature OSS project for this or if most teams build their own lightweight gateways.
More from coding & agent
- Open-Weight Kimi K3 Rolls Out in GitHub Copilot for Cost-Effective Agentic Coding — 0xkarasy · 2026-08-07
- Managing Tens of Thousands of AI Agents: CTO.new Cuts 95% of Costs with Turso — glcst · 2026-08-07
- Happenstance 2.0 Launches: AI Agent for Precision Social Network Search — var_epsilon · 2026-08-07
- Optimizing AI Coding Context: On-Demand Skills to Slash Token Tax — AlexKim · 2026-08-07
- Cutting CLAUDE.md by 87%: A Context Diet Framework with Real Numbers — AlexKim · 2026-08-07
- Triage Your AI Coding Context: A 4-Bucket Cleanup Strategy — AlexKim · 2026-08-07