Hugging Face's datatrove 0.10.0 adds HF Jobs pipeline executor, no Slurm needed
vanstriendaniel · x · 2026-08-14
Hugging Face's open-source data processing library datatrove (used to build FineWeb, FineWeb2, FinePDFs) released version 0.10.0. Key updates:
- JobsPipelineExecutor: Run pipelines on Hugging Face Jobs with fan-out, multi-stage dependencies, retries, and resume, without needing a Slurm cluster.
- HF storage buckets as DataFolder: Read, write, and log directly to hf://buckets/... paths.
- Reasoning outputs preserved in inference results.
Install with pip install datatrove[io,processing].
The project has 3.3k stars on GitHub, providing platform-agnostic pipelines for large-scale text data processing, supporting local and Slurm execution.
More from Infra
- RSI to enter enterprises: frontier models train specialized models, huge infra needs — morgymcg · 2026-08-14
- Agentic AI Cools Off, but Infrastructure Spending Keeps Climbing — YvesMulkers · 2026-08-14
- Agent Auto-Generates Earnings Reports: Cloudflare Case Shows AI Traffic Inflection — draecomino · 2026-08-14
- Hosting a 3T Model on Cerebras Requires 68 Chips and 1.5MW — zephyr_z9 · 2026-08-14
- $1T of AI Infra Buildout to Solve $1M Math Problems? — suchenzang · 2026-08-14
- Intel and MiniMax Release MXFP4 Quantized Model, Topping FP4 Benchmarks — HaihaoShen · 2026-08-14