pigzpp: C++ Rewrite Achieves 50x Faster Parallel Compression Library
jedisct1 · x · 2026-08-27
The paper introduces pigzpp, a from-scratch C++23 rewrite of the widely deployed pigz parallel gzip utility.
- Design Overhaul: The original pigz relied on process-global mutable state, making it non-reentrant and unsuitable for direct embedding. pigzpp transforms this into a thread-safe library with a single accelerated DEFLATE core.
- Multi-language Support: The core is exposed to C++, Python, WebAssembly, Go, and Rust. Built on this same core, it also provides application-level conveniences like multi-entry ZIP archives and a fast PNG codec.
- Performance Gains:
- The portable zlib-ng backend preserves gzip/zlib compression ratios.
- The Intel ISA-L backend (x86-64 only) serves as a fast path, producing about 10% larger output at level 6. In benchmarks on a text corpus, zlib-ng achieves 2.1x the CLI throughput of pigz and 16x that of Python's in-memory gzip; ISA-L reaches 8.5x and 50x respectively.
- Compatibility: Outputs remain standards-compliant and cross-decode with gzip, pigz, and unzip.
The result is a self-contained, multi-platform compression stack and a case study in AI-assisted modernization under automated compatibility tests.
Related event: Microsoft Open-Sources pigzpp, a C++23 Parallel gzip Rewrite(2 posts)→
More from Infra
- Qwen3.8-Flash-Next Fits on a 48GB MacBook: Pruning + SSD N-gram Table, 39GB RAM — EyalToledano · 2026-08-27
- Nvidia earnings suggest AI inference compute is not the bottleneck — pstAsiatech · 2026-08-27
- A $899 Mac mini running a local 8B model ended my ChatGPT usage-cap headaches — ugcfast · 2026-08-27
- Grokpute launches distributed GPU training network — jw2yang4ai · 2026-08-27
- Weaviate ships query profiling: one flag pinpoints slow-query bottlenecks inline — victorialslocum · 2026-08-27
- Storage architecture for AI sandboxes: Local root + S3 — aniketmaurya · 2026-08-27