HF engineer's agent hits 99.7% storage savings on Parquet edits in DuckDB via content-defined chunking
lhoestq · x · 2026-11-02
Hugging Face engineer Quentin Lhoëst reports that his agent achieved up to 99.7% storage savings on Apache Parquet edits in DuckDB, via a draft PR (#27911) adding content-defined chunking to the Parquet writer.
- Why it matters: with fixed-size data pages, editing one value shifts every later page boundary in a column chunk, so a deduplicating uploader (like the Xet-based HF client) sees the entire tail as new bytes. CDC cuts pages where content says instead.
- Compatibility: files remain ordinary Parquet — same page types and encodings, readable by any reader.
- Usage: COPY t TO 'f.parquet' (FORMAT PARQUET, CONTENTDEFINEDCHUNKING true), with optional minchunksize: 262144, maxchunksize: 1048576, normlevel: 0.
- Benchmarked on 150 MB, 2M-row files (four columns incl. a 64-char text column) using the Xet chunker/hasher across append/update/insert edits.
More from coding & agent
- Grok Bot template roundup: Home Assistant watchdog, Quarto blog shipper, X unfollow tool — RachelVT42 · 2026-11-02
- Harnis: an open-source call-level seatbelt for local stdio MCP servers — L00pLegend · 2026-11-02
- RAGFlow: BM25+grep alone hits 91% FRAMES on MiniMax M3, no vector index — dair_ai · 2026-11-02
- A 144M model drives most NPC choices in a village sim that learns on a 6GB laptop GPU — Whoa_damn_ · 2026-11-02
- AI favors engineers who care about outcomes over the craft itself — brada · 2026-11-02
- Claude builds an explainer-video workflow from scratch in Managed Agents — brada · 2026-11-02