Hugging Face launches Buckets for hosting massive 33TB files
NielsRogge · x · 2026-08-23
Hugging Face released a new repository type called 'Buckets', allowing users to share and host extremely large files, marking the fourth type alongside models, datasets, and Spaces.
Key Features:
- Optimized for large files: Powered by Xet technology, it provides S3-like object storage optimized for AI workloads.
- Non-Git based: Unlike traditional Git repos, Buckets are remote object storage containers that do not track file history, making them suitable for storing training checkpoints, logs, intermediate artifacts, or large file collections that don't need version control.
- Deduplication & Speed: Supports content-addressable deduplication storage.
Use Case:
- @rerundotio released xdofai's ABC-130k bi-manual teleoperation dataset. The converted RRD files are hosted in a Bucket totaling 33TB, which users can visualize in Rerun or use directly in training workflows.
More from Infra
- Executor Gateway Cuts Tool Context Usage from 278k to 1k Tokens — ethanniser · 2026-08-23
- Agent-Aware Infra: Optimizing Inference via Cache and Scheduling — _ScottCondron · 2026-08-23
- NVIDIA Feynman Interconnect: Solving Data Bottlenecks in Trillion-Parameter Training — BenBajarin · 2026-08-23
- Energygraph 1.3: Monitor GPU/CPU Power Usage in Terminal — mazarax · 2026-08-23
- Supporting Linux on Metal Could Extend Apple's Lead in Consumer AI Hardware — StefanoGogioso · 2026-08-23
- Test your GPU undervolting stability by running a model training task — Think_Wing_1357 · 2026-08-23