HF Jobs Built-in Serving Boosts Throughput 47x
vanstriendaniel · x · 2026-07-13
The author shares an underrated trick for **Hugging Face Jobs**: **instead of just running scripts in a Job, serve the model directly inside the Job**. By putting a **sglang server + async driver** into the same `uv` script and launching it via a Hub URL, they leveraged **continuous batching** to boost throughput on a single GPU from **3.2x realtime to 47x**. This technique isn't limited to Apollo 11 audio processing; the author believes any scenario involving "batch processing a massive pile of files through a model" can benefit. The use case involved using a **0.9B open-source model, MOSS-Transcribe-Diarize**, for transcription, speaker diarization, and timestamping: - Processed **174 hours** of Apollo 11 mission audio - Total cost: **$9.46** - Completed 103 audio segments in one go - Generated **45,355** timestamped speaker segments The final result transformed old NASA recordings into a searchable database where you can query "who said what and when."
More from Infra
- Fluidstack raises $830M at $7.5B valuation as Anthropic backs a $50B compute buildout — rohanpaul_ai · 2026-07-21
- Early Krea2 Gradio WebUI targets 6GB low-VRAM local runs — Fluid_Kaleidoscope17 · 2026-07-21
- Z.AI starts running a 1GW AI data center built entirely on domestic chips — Polymarket · 2026-07-21
- Local models feel far more capable once paired with the right harness — Soft-Barracuda8655 · 2026-07-21
- Voice-agent teams should use platforms first, then own STT events when failures get weird — FollowingSuitable941 · 2026-07-21
- Alibaba open-sources T-Head SAIL, the software stack behind its AI chips — 量子位 · 2026-07-21