When to move from a homegrown FastAPI LLM proxy to a proper gateway
Kev926 · reddit · 2026-11-10
An engineer describes outgrowing a homegrown FastAPI LLM proxy that originally let apps switch between OpenAI and Anthropic without code changes. With 4 apps across 3 teams sharing two API keys, pain points now include team-level keys and usage limits, proper request tracking, keeping sensitive customer data out of prompts before they reach providers, and logging that's just a basic Postgres table. They ask whether a dedicated LLM gateway is worth the switch — a familiar inflection point for engineering teams scaling multi-tenant LLM infrastructure.
More from coding & agent
- verity-gate 0.2.0 ships its agent action-vetting gate as MCP servers for pip and npm — W1RELESS · 2026-11-10
- HuggingFace SafeAgents: a safety layer between your harness and your model — blaizedsouza · 2026-11-10
- rakyll teases Agent Substrate as 'agentic compute fabric' ahead of KubeCon keynote — rakyll · 2026-11-10
- Developer uses a 1.9B decision model on a laptop to flag dirty restaurant tables in real time — JosephJacks_ · 2026-11-10
- Microsoft FORGE Lab uses agentic AI tools to find memory-safety bugs in Rust core libs — jedisct1 · 2026-11-10
- CTM-AI: open-source multi-agent framework inspired by consciousness theory, NeurIPS 2026 paper — pliang279 · 2026-11-10