ModelSphere launches as open-source LLM serving platform for mixed GPU fleets
Codeblix_Ltd · reddit · 2026-10-17
Phancy Group and RiseUnion launched ModelSphere, an Apache 2.0 open-source LLM deployment platform on GitHub targeting production serving rather than a new model or chat app.
- Built on Kubernetes, Helm, SGLang and vLLM
- Claims autoscaling, cache-aware request routing, shared KV cache, multiple accelerator families, and recovery of in-flight generations after interruptions (project claims, no independent benchmarks)
- Quick start requires an existing Kubernetes cluster plus kubectl, Helm, helmfile and helm-diff
Aimed at teams wanting a control plane for model serving, at the cost of another operational layer; community reception is still early.
More from Infra
- vLLM makers Inferact partner with AMD to optimize agentic inference on Instinct GPUs — woosuk_k · 2026-10-17
- Right-sizing context windows: Gemma 4 26B voice app fits on a single H100 — Ctbhatia · 2026-10-17
- PyTorch now runs natively on AWS Trainium with zero code changes — PyTorch · 2026-10-17
- Zuck bets most inference runs locally: Meta emerging as the champion of frontier local AI — StefanoGogioso · 2026-10-17
- Flux 3 Klein benchmark: nvfp4 hits 13.7s per image vs 25.2s for fp8 — dh7net · 2026-10-17
- .NET 10 Lambda ARM64 base image ships with 170MB of leftover Yum cache — DanielLockyer · 2026-10-17