Serving a Full Coding Model Portfolio on One 8x B200 Node With ~1-Minute Switching
josh_wills · x · 2026-10-02
Cedana, working with NVIDIA's Dynamo team, shows how to host a portfolio of frontier coding models on a single 8x B200 node, switching between them in about a minute with in-flight sessions intact — solving the 9-34 minute cold-start problem that makes model portfolios impractical.
Key points:
- Enterprises increasingly run model portfolios: cheap small models for easy tasks, frontier models for the hard tail. The bottleneck is GPU utilization — models don't fit on one node together, and cold starts kill switching.
- Based on TraceLab, a public trace of 4,300 Claude Code and Codex sessions: median requests finish in 38 seconds, but the 6% running past 10 minutes consume 69% of all request time — heavily long-tailed traffic.
- The stack: NVIDIA Dynamo serves, NeMo Switchyard routes and escalates when smaller models fail, Cedana handles 1-minute model switching. Coding agents connect to a single API endpoint.
- The article lists the portfolio and VRAM footprints, e.g. DeepSeek-V4-Pro-NVFP4 at 873 GiB for long-horizon planning.
Related event: Cedana and NVIDIA host full coding model fleet on one node(2 posts)→
More from coding & agent
- Generic API-catalog MCP server vs one tool per job: a context-bloat experiment — MandeepKaur9407 · 2026-11-16
- Berkeley paper: LLMs answer with stale values even when the update is in context — ombre-ai · 2026-11-16
- Effective HTML: 4.2k-star agent skills turn wireframes into self-contained interactive prototypes — tom_doerr · 2026-11-16
- skills-gate: one gate skill fronts 148 agent skills at just 389 tokens — Distinct-Pie2389 · 2026-11-16
- Open-weight Karpo-Decision-31B tops Decision Index at 73.9, one forward pass per answer — CodeByPoonam · 2026-11-16
- shadcn boosts @uisdk: one-shot UI rebuild that's instantly E2E-testable by AI — shadcn · 2026-11-16