Olla v0.1.0: open-source LLM proxy unifies Ollama, vLLM, SGLang with failover
2shanigans · reddit · 2026-11-03
Olla, a lightweight Go proxy and load balancer for homelab and on-prem LLM inference, hit v0.1.0 (its 25th release). It unifies mixed backends—Ollama, vLLM, SGLang, llama.cpp, LM Studio, MLX and more.
Highlights:
- Model-aware routing with automatic discovery across backends
- Load balancing (priority, round-robin, least-connections) with health checks and circuit breakers
- Unified OpenAI- and Anthropic-compatible APIs with protocol translation, working with Open WebUI and Claude Code
- Sticky sessions for KV cache locality
- Built-in dashboard, request stats and Prometheus metrics; single binary or Docker, no database
v0.1.0 adds Hugging Face TEI support (embeddings/reranking), a reworked dashboard, configurable retry/failover including non-idempotent requests, better capability detection, and concurrency fixes. The dashboard screenshot shows 6 healthy endpoints, 181ms average latency and 90.1% sticky-session hit rate.
More from Infra
- Apple made 2x the free cash of Microsoft, Meta, Alphabet and Amazon combined last quarter — asymco · 2026-11-03
- Huawei tells customers to source AI chip memory themselves; smuggled HBM3E hits ~$4,000 — The Decoder · 2026-11-03
- RTX 5090 with 64GB and 96GB VRAM 'confirmed' by US GPU repair tech — MundanePercentage674 · 2026-11-03
- Samsung's second Taylor fab fully booked for AMD and NVIDIA HBM base dies before construction even begins — zephyr_z9 · 2026-11-03
- QLink insight: one model's store is already a valid memory write for another — zeeshanp_ · 2026-11-03
- All six collectives on 4 QwenGPUs match real NCCL bit-for-bit on GB200s — zeeshanp_ · 2026-11-03