Photoroom reclaims ~1,000 idle GPU hours a week with SkyPilot across Slurm and Kubernetes
skypilot_org · x · 2026-11-12
Photoroom engineer David Briand details how SkyPilot unified the company's GPU fleet and turned idle inference GPUs into usable capacity.
- In October alone, researchers ran about 1,000 GPU hours a week on inference GPUs that used to sit idle off-peak
- A single researcher with an agent can use every GPU without engineer involvement: dataset prep on local RTX 3090/RTX 6000 Ada, diffusion model training on 128 H200s over InfiniBand, then checkpoint evals on idle inference GPUs with a demo shared to the product team
- SkyPilot unifies Slurm (including the H200 cluster), multiple Kubernetes clusters, SSH-only machines, and on-demand GPUs from several providers — workloads spill over to external capacity with no procurement or new setup
- The main training cluster stays reserved for large trainings that truly need it
The team evolved from scattered office workstations (Slack full of "is node X free?") to Slurm + a racked server room, then a cloud H200 cluster, and finally unified orchestration. With GPUs still scarce, every reclaimed hour matters.
Related event: Photoroom Uses SkyPilot to Recover 1,000 Idle GPU Hours Weekly(2 posts)→
More from coding & agent
- Ora's Ranker 3.0 grades whether agents can actually use your site; scores drop 40+ points — yenkel · 2026-11-12
- Gemini API Managed Agents now support self-hosted sandboxes, incl. Vercel — steren · 2026-11-12
- Rayline launches ARC, a per-turn model router claiming 90% cost cuts — testingcatalog · 2026-11-12
- Karpathy's llm-wiki: a pattern for building personal knowledge bases with LLM agents — stochasticchasm · 2026-11-12
- Agent self-improvement with frozen weights: failure traces add just 4pp, blind search gets 60% — burny_tech · 2026-11-12
- Microsoft on agentic schedulers: LLM proposes, deterministic code validates — WirelessLife · 2026-11-12