HybridInfer: open-source router runs local models first, auto-falls-back to cloud on stalls
simrankoulsm · reddit · 2026-09-02
A developer released HybridInfer, an Apache-2.0 reliability-aware inference router that solves the pain of local models silently stalling or OOM-ing on long prompts:
- Every request goes to the local model (Ollama) first; if it stalls (no token for N seconds), OOMs, or errors, it falls back to a remote model within the same request.
- Learns which prompts (usually long ones) choke your machine and routes them out up front after seeing them fail.
- Pulls a wedged model out of rotation and probes it back after a cooldown.
It's an OpenAI-compatible server, so any compatible client gets local-first + automatic fallback for free; remote can be OpenAI, OpenRouter, a vLLM box, etc. It orchestrates only — it doesn't run weights itself. A Kotlin/Android version exists for on-device apps.
Install:
pip install hybridinfer
hybridinfer init
hybridinfer serve
Early v0.1; the author wants feedback on the fallback heuristics from people running local models daily.
More from coding & agent
- LLM one-shots a Minecraft 3x3 piston door by building its own Linux test infra — Vjeux · 2026-10-18
- Dev builds Mumbai-inspired 3D manga background generator from OpenStreetMap data — CurieuxExplorer · 2026-10-18
- Solo founder builds a money-printing data service on Claude's Pi Durable harness — MicahBerkley · 2026-10-18
- Dev builds Arch-based OS with custom desktop and browser using Claude Code — Snoo-10493 · 2026-10-18
- Five SaaS agent workflows fail the same way — and it's never model intelligence — usehive · 2026-10-18
- Claude skill generates hand-drawn interactive diagrams that teach kids how things work — CatAstro_Piyush · 2026-10-18