hf-mem 1.0.0 nears stable release with vLLM/SGLang-aligned memory estimates
anton_lozhkov · x · 2026-11-10
alvarobartt previews hf-mem 1.0.0 (stable): more accurate memory estimates, faster metadata reads, and new runtime overhead estimations better aligned with vLLM and SGLang. A preview estimates Qwen3.8-27B (BF16, 32k context, batch 1) at 63.26 GiB total — 51.75 GiB params + 2.30 GiB KV cache + 9.22 GiB overhead — fitting 71% of one 96 GiB GPU and supporting up to 262,144 tokens.
More from Infra
- Gary Marcus slams Nvidia after report says Commerce inquiry was dropped — GaryMarcus · 2026-11-10
- Claude Fable 5.2 and Mythos 5.2 spotted on AWS Bedrock, priced 20% below 5.1 — testingcatalog · 2026-11-10
- e-means paper trick: IVF approximate assignment cuts clustering cost 34x at 10B vectors — antoine_chaffin · 2026-11-10
- Telekom, NVIDIA and SAP expand German AI cloud, boosting capacity 50% — NinaDSchick · 2026-11-10
- Two-person YC-backed Archgen wins chip layout contest against 131 entries — testingcatalog · 2026-11-10
- NVIDIA Dynamo in 5 minutes: OpenAI-compatible frontend enables backend hot-swap and KV cache reuse — NVIDIA Developer · 2026-11-10