Self-hosted AI platform: 3 old GPUs, vLLM per card, and a semantic router as one endpoint
JeremyCMorgan · x · 2026-11-05
Techno Tim published a detailed writeup/video of his self-hosted AI platform: one Linux box with an RTX 3090, a 3060, and an RTX 2000 Ada, each running vLLM, plus a semantic router so Copilot, Open WebUI, and Home Assistant all see a single endpoint.
- Problem solved: with multiple GPUs, models, and apps, he didn't want to pick a machine and model per request or update every app when switching models.
- Approach: route by request type — light requests to the 3060, the bigger model on the 3090, embeddings and speech on dedicated hardware; clients hit one OpenAI-compatible endpoint.
- The article covers how the pieces connect and why, and shows how to build a smaller version without his Ansible config; no tok/s or wattage numbers yet.
More from coding & agent
- LMArena opens Agent Arena for agent evaluation — arena · 2026-11-05
- Running Qwen 27B Q4 at 0.7 tok/s on a 4GB-VRAM laptop as a main coding model — Felix_455-788 · 2026-11-05
- His agent now auto-labels his Google Calendar in the background — steren · 2026-11-05
- The security startups racing to rein in AI agents — pstAsiatech · 2026-11-05
- OpenAI October dev roundup: Decisions API, 12-hour caching, 8x faster Codex tokens — OpenAIDevs · 2026-11-05
- Datadog previews Canvas, a new way to collaborate with AI agents on observability — farooqib · 2026-11-05