huggingface_hub v1.32 dedupes Xet files in local cache: 5 repos sharing 20GB weights now use ~20GB
victormustar · x · 2026-09-18
huggingfacehub v1.32 brings content-addressed deduplication to the local cache: identical Xet-backed files across repos are stored only once. Example: 5 repos sharing the same 20 GB weights file now take 20 GB on disk instead of 100 GB.
- Xet already made remote storage content-addressed; that identity now extends into the local cache
- Once a blob is local, other repos can reuse it with zero payload download
- A real disk saver for anyone pulling many fine-tuned or quantized variants built on shared base weights
Related event: huggingface_hub v1.32 Adds Shared Download Cache Across Repositories(2 posts)→
More from Infra
- 28 hashes logged: crowd audits Gensyn's open-1b, first verifiable training run — benfielding · 2026-09-19
- Astonishing wealth transfer: hyperscalers' free cashflow halved, mostly to Nvidia — docmilanfar · 2026-09-19
- Token Superposition Training Cuts Pretraining Compute 2.5x on 10B MoE — gordic_aleksa · 2026-09-19
- Groq founder flags slowing US power production growth as AI buildout risk vs China — TheZachMueller · 2026-09-19
- Université Laval builds new AI-dedicated HPC cluster with Mila and Vector Institute — Mila_Quebec · 2026-09-19
- Virginia, world's largest data center hub, unveils new restrictions on large projects — Polymarket · 2026-09-19