Xet Dedup Saves Hugging Face 80% of Storage Globally, Engineers Say
lhoestq · x · 2026-10-22
Detailing the storage behind the new hf:// DuckDB path: Hugging Face Buckets are built on Xet, and global deduplication saves 80% of storage. The same file takes less space on HF than on any other filesystem or cloud provider — a key reason the hf:// workflow is efficient.
Related event: Hugging Face Ships DuckDB 2.0 Extension for SQL Access to Datasets(3 posts)→
More from Infra
- Black October: LiteLLM credential stealer, npm worm and thousands of poisoned GitHub workflows — AIFlow_ML · 2026-10-23
- July 28 MCP release makes remote servers stateless, biggest revision since launch — bibryam · 2026-10-23
- MCP keynote: AAIF hits 305 members, Skills over MCP and Tasks/Events roadmap revealed — bibryam · 2026-10-23
- Only 8% of cloud vendors support OIDC — dev calls to kill long-lived API keys — cramforce · 2026-10-23
- Swift-1.5-Qwen3.8-27B ported to hipfire MQ4: 2.2x faster end-to-end than GGUF on a 7900 XT — Queasy_Asparagus69 · 2026-10-23
- Intel CTO: chip output can at best 2X while AI compute grows 3X per year — SammaVaco · 2026-10-23