Author releases dedup_estimate.py gist to reproduce the Parquet dedup benchmark
lhoestq · x · 2026-11-02
Quentin Lhoëst published a gist, dedupestimate.py, to reproduce the benchmark numbers behind his DuckDB content-defined chunking work.
- The script uses DuckDB COPY into nxet:// via nanoxet, writing v1/v2 versions of the same data across append, update, and insert edit types, with SEED=17 for reproducibility.
- It complements DuckDB PR #27911, letting anyone estimate the real storage savings after Xet deduplication for a given edit.
More from Infra
- Lumentum deal sparks debate: does the sale signal doubts about the AI cycle lasting? — BenBajarin · 2026-11-03
- Rohan Anil's CoreAuto trains MoE across two global clusters over a 25 Gbps link — _arohan_ · 2026-11-03
- Anthropic Q3: $17B revenue but swing to $567M operating loss, $895B compute commitments — rohanpaul_ai · 2026-11-03
- AI compute as a planet: three quarters under the US flag, China one seventh — LeviTurk · 2026-11-03
- Custom Qwen3.8-27B engine doubles llama.cpp speed on a 3090 with exact greedy outputs — Nice_Profession_9078 · 2026-11-02
- Uncensored DeepSeek-V4.1 quants cut refusals from 85% to 0% on 12 GB VRAM — gladkos · 2026-11-02