ngrok launches AI Gateway: route self-hosted and frontier models via one URL
anthara_ai · x · 2026-09-05
ngrok released an AI gateway at ngrok.ai with the pitch "Never bet on one model": change your baseURL to gateway.ngrok.ai, swap in an ngrok key, and route between self-hosted local models and frontier providers (OpenAI, Anthropic, etc.) through one managed endpoint.
Key points:
- Local LLMs over private connectivity: route to self-hosted models (demo uses an unsloth/Qwen3.5 GGUF) without public IPs or inbound ports, with fallbacks like gpt-5.4 or claude-opus-4-6.
- BYOK: plug in existing provider keys, pay providers directly at current rates, manage everything in one place.
- Scoped access keys per app or developer, restricting which providers/models each can call.
- Observability: tokens, latency, errors and costs rolled up per app, dev, and model — something provider dashboards don't show.
- Compatible with OpenAI SDK, Anthropic SDK, and Vercel AI SDK.
Related event: ngrok launches AI Gateway: one URL to route cloud and self-hosted models(7 posts)→
More from Infra
- GLM-5.3-Flash NVFP4 benchmarked: 2x H200 matches 4x RTX Pro 6000 on throughput — TheZachMueller · 2026-10-21
- DeepSeek V4.1 Flash inference teardown: Engram cuts 30% params, WideEP spreads 552B over 64 GPUs — teortaxesTex · 2026-10-21
- llm-d 1.0 ships: multi-tenant QoS, zero-downtime updates, prefix-cache-aware routing — _ScottCondron · 2026-10-21
- DeepInfer math: 1GW of GB200 could generate $100B+ revenue at DeepSeek's current prices — bookwormengr · 2026-10-21
- FastVideo releases FastH3 Omni weight collection with FP8 and NVFP4 ComfyUI builds — multimodalart · 2026-10-21
- Voice models should run on-device: Phonon open-source release sparks debate — timshi_ai · 2026-10-21