HybridInfer: open-source router runs local models first, auto-falls-back to cloud on stalls
simrankoulsm · reddit · 2026-09-02
A developer released HybridInfer, an Apache-2.0 reliability-aware inference router that solves the pain of local models silently stalling or OOM-ing on long prompts:
- Every request goes to the local model (Ollama) first; if it stalls (no token for N seconds), OOMs, or errors, it falls back to a remote model within the same request.
- Learns which prompts (usually long ones) choke your machine and routes them out up front after seeing them fail.
- Pulls a wedged model out of rotation and probes it back after a cooldown.
It's an OpenAI-compatible server, so any compatible client gets local-first + automatic fallback for free; remote can be OpenAI, OpenRouter, a vLLM box, etc. It orchestrates only — it doesn't run weights itself. A Kotlin/Android version exists for on-device apps.
Install:
pip install hybridinfer
hybridinfer init
hybridinfer serve
Early v0.1; the author wants feedback on the fallback heuristics from people running local models daily.
More from coding & agent
- Integrating Grok Bot into Superhuman via MCP: A 4-layer architecture — bfrench · 2026-09-02
- Qwen3.8-Max-0902 tops CodeArena WebDev with record score and pricing — Alibaba_Qwen · 2026-09-02
- Agentic engineering is fundamentally constraint engineering — burny_tech · 2026-09-02
- Smarter models overthink everything, slowing down coding workflows — sandyyevans · 2026-09-02
- InternReviewer Uses RL to Improve Peer Review Accuracy — Shanghai-AI-Laboratory · 2026-09-02
- Control-data flow separation keeps prompt optimization from breaking multi-agent pipelines — UWaterloo · 2026-09-02