Inference Stack Launches One-Command Self-Hosted LLM Serving with vLLM, LiteLLM and llm-d
AccBalanced · x · 2026-11-22
BerriAI has released Inference Stack (v0.1.2 preview), an open-source bundle that deploys a complete self-hosted LLM inference setup with a single command, under the pitch of "owning your inference".
- Deployment: supports Docker Compose (single machine) and Kubernetes; a one-line curl | bash installer sets everything up, then you log in at localhost:8080 with a generated admin password.
- What's inside:
- vLLM runs models on your GPUs, with Hugging Face search and GPU-requirement checks built into the launcher;
- LiteLLM exposes a single API for requests, with key creation, team/budget management and a Playground;
- llm-d routes each request to a vLLM replica using cache and queue data, on by default with adjustable weights;
- Grafana + Prometheus dashboards show model health, traffic and latency.
- Real models need NVIDIA GPUs; a simulation mode provides canned responses for GPU-less trials.
More from coding & agent
- Cross-provider adversarial review: hedge shared model blind spots — craigbalding · 2026-11-22
- Users complain GPT-5.5 writes bloatware and duplicates functions — yacineMTB · 2026-11-22
- Easy MCP: open-source tool lets web AI read and edit files on your PC — Stunning_Pack_4636 · 2026-11-22
- The only software that survives is what agents can use fast: frictionless onboarding as survival criterion — alexmacgregor__ · 2026-11-22
- Are Gemini 4 Neon fans actually using it in Antigravity? Coder asks — iScienceLuvr · 2026-11-22
- '2,000 lines from Claude, 10-second Codex review': devs fret over AI engineering quality — _jaydeepkarale · 2026-11-22