Hugging Face Engineer Cuts Parquet Storage by 99.7% with Content-Defined Chunking

Hugging Face engineer Quentin Lhoëst achieved up to 99.7% storage savings on Apache Parquet edits in DuckDB using content-defined chunking, and released the dedupestimate.py script so others can reproduce the benchmark results.

2026-11-02 ~ 2026-11-02 · 2 related posts

• HF engineer's agent hits 99.7% storage savings on Parquet edits in DuckDB via content-defined chunking — lhoestq · 2026-11-02
• Author releases dedup_estimate.py gist to reproduce the Parquet dedup benchmark — lhoestq · 2026-11-02