Hugging Face datatrove 0.10.1 fixes empty-doc crashes and a silently ignored skip parameter
vanstriendaniel · x · 2026-09-30
Hugging Face's data processing library datatrove shipped v0.10.1, a bug-fix release:
- Stats fixes: DocStats, LineStats, ParagraphStats, SentenceStats, WordStats and WordsContaminationStats no longer raise ZeroDivisionError on empty, whitespace-only or punctuation-only documents; ratios/averages are recorded as 0.0 and included in summaries. GopherRepetitionFilter no longer crashes on empty documents.
- Behavior fix: HuggingFaceDatasetReader's documented skip parameter, previously ignored, now skips the first skip documents with text in each task; generated ids are unaffected by skip. limit and skip are per-task on all readers.
- Engineering: the test suite adds Hypothesis property tests, and CI is faster.
The maintainer notes some fixes can change output for runs that previously crashed or ignored a parameter, while previously working runs produce identical results.
More from coding & agent
- 1,000 AI agents discover new CRISPR-like system in virus DNA within 24 hours — CurieuxExplorer · 2026-09-30
- SEAD: SAGE defender cuts tool-agent attack success from 48% to 4% against DART attacks — Xinjie Shen · 2026-09-30
- IBM's Q&D trains proactive agents to ask better questions, beating a 15x larger model — ibm · 2026-09-30
- One owner per decision: managing coding-agent project memory across eight repos with Git — oliver-zehentleitner · 2026-09-30
- Hister: open-source private search engine turns your pages and files into an MCP-searchable agent index — bibryam · 2026-09-30
- Monid, 'OpenRouter for Agent Tools,' Unifies 2,000+ Tools Across 72+ Providers — aigclink · 2026-09-30