Hugging Face demos Xet incremental uploads: syncing a 100 MB dataset sends only ~6 MB
Hugging Face · youtube · 2026-10-07
An official Hugging Face walkthrough shows how the Xet storage layer uploads only changed chunks when datasets evolve:
- CSV: appending 100 rows to a 191 MB NYC taxi CSV and re-uploading transmits only the new pieces via Hugging Face Buckets.
- Parquet: with PyArrow 21+'s usecontentdefinedchunking=True, prepending 100 rows to a 52.9 MB file sends just 8.6 MB over the wire; on a 100 MB file, upload drops from 90 MB to 6 MB.
- Workflow: creating Buckets, cp and sync with the hf CLI.
- Production gotchas: S3 tool compatibility, versioning, sorting, and writer settings.
Aimed at data/ML engineers keeping Hub datasets up to date; the companion blog includes full commands and code.
More from Infra
- Rumor: chip startup Terafab in talks with all three logic foundries, Samsung ahead — pstAsiatech · 2026-10-07
- Adaption AI launches AutoScientist Leaderboard ranking custom models across 44 domains — sarahookr · 2026-10-07
- CoreWeave enters India with 240 MW AdaniConneX deployment in Navi Mumbai — ayushthakur0 · 2026-10-07
- Qdrant squeezes EmbeddingGemma 2 vectors to 0.4GB from 30GB, keeping 94.5% quality — qdrant_engine · 2026-10-07
- Qdrant compresses Google EmbeddingGemma 2 vectors 77x with only 5% quality loss — qdrant_engine · 2026-10-07
- ASML tipped to ship 120-125 EUV tools in 2028 as capacity expansion accelerates — zephyr_z9 · 2026-10-07