Parquet Java adds content-defined chunking to enable byte-identical page dedup
lhoestq · x · 2026-10-14
A draft PR (#3818) on apache/parquet-java brings content-defined chunking (CDC) of data pages, matching capabilities already in Parquet C++ and parquet-rs.
- Pages end at content-defined boundaries, so files sharing runs of values produce byte-identical pages that content-addressable storage like Hugging Face Xet can deduplicate
- It's a port of the Arrow C++/arrow-rs chunker with identical gear tables, mask derivation and cut rules, so page boundaries match C++
- Adds CdcOptions (256 KiB min / 1 MiB max / normalization 0 by default) and writer options like withContentDefinedChunking()
- Iceberg Java needs it to honor write.parquet.content-defined-chunking. table properties; opt-in, still in draft
More from Infra
- OpenJet claims Qwen3.8-27B coding runs 1.9x faster than llama.cpp on one RTX 3090 — JeremyCMorgan · 2026-10-23
- Jensen Huang lays out NVIDIA's full Vera Rubin agentic system: NVL72, MGX, CMX and RTX — rohanpaul_ai · 2026-10-23
- ONNX to Transformers.js: running models directly in your browser — JosephJacks_ · 2026-10-23
- MiniMax M3.1 Flash free for 3 days on 0G, with 1M context and 512K output — MiniMax_AI · 2026-10-23
- "We bet on Cloudflare and it paid off": R2 and D1 at billion scale — dinasaur_404 · 2026-10-23
- Public data is drying up; private training data is turning into a browsable catalog — SucceededMind · 2026-10-23