pigzpp: Fast, Parallel, Portable Compression for the Whole Stack
Thamme Gowda
cs.DC
2026-08-25
A C++23 rewrite of pigz as a thread-safe library with zlib-ng and ISA-L. On 128 MB text at level 6, portable zlib-ng is 2.1× pigz; x86-64 ISA-L is 8.5× with ~9% larger files.
gzip is still the default: .gz files, DEFLATE inside ZIP, PNG IDAT, HTTP Content-Encoding, Docker/OCI layers. The format was designed for one core. Mark Adler's pigz made it parallel by splitting input into blocks, and the CLI has been the default way to saturate a multi-core box with compatible gzip.
pigz is a program, not a library. About sixty mutable fields live in a process-global struct g. Adler said so on Stack Overflow: "pigz is not a library." Callers today are more often Python pipelines, Go services, or a browser WASM module. Each language then re-solves parallel gzip on its own (pgzip in Go, gzp in Rust), so the same speedup is written five times.
Thamme Gowda (Microsoft; personal project) rewrote the design in C++23 as a reentrant library with no global state. Configuration is passed in; threads are C++20 std::jthread; the hot path has no global lock.
The parallel plan is pigz's. Default 128 KiB blocks go to a pool. Each worker primes itself with the last 32 KiB of the previous block so cross-block back-references, and the ratio, survive. CRC-32 is computed per block and folded with crc32combine in logarithmic time. A single writer emits the gzip header, blocks in input order, and the trailer. gzip and pigz can decode the result.
Two statically linked DEFLATE engines sit behind one API:
The same core is bound to C++, Python (nanobind, one abi3 wheel for CPython 3.12+), WASM (Emscripten, SIMD and SharedArrayBuffer threads), Go (cgo), and Rust (FFI). ZIP and PNG are application layers: ZIP sends each member through the parallel compressor; PNG filters scanlines and writes IDAT.
Much of the rewrite was done by coding agents under human direction. Acceptance rested on lossless round-trips, cross-decoding with gzip/unzip, a test suite, and repeatable benches. The author's own lesson: agents move fast when oracles are mechanical; the hard part was untangling global state.
Native numbers are from Ubuntu 22.04 WSL2 on an Intel Xeon W-2235 host. The guest saw five cores and ten logical CPUs. Unless noted, the corpus is 128 MB of English and Chinese Wikipedia, level 6, eight workers; one warm-up plus seven timed runs, median reported.
CLI (Figure 1):
| Method | Throughput | vs pigz | Ratio |
| gzip | 17 MB/s | 0.13× | 2.83 |
| pigz | 133 MB/s | 1× | 2.83 |
| pigzpp zlib-ng | 285 MB/s | 2.1× | same as pigz |
| pigzpp ISA-L | 1124 MB/s | 8.5× | 9% more output |
Language bindings, same corpus:
On the largest python:3.12 image layer (637 MB), stdlib gzip took 19.6 s. zlib-ng reached 381 MB/s at ratio 2.83, next to pgzip at 376 MB/s / 2.76, about 11× stdlib. ISA-L was about 42×, 0.5 s. That is the compression stage, not an end-to-end docker build.
PNG on the 24-image Kodak set (768×512 RGB): the ISA-L fast preset is about 11.8× Pillow's default and 1.6× OpenCV, at similar size. ZIP writes are about 12× Python zipfile at the same ratio on zlib, up to 28× with ISA-L. Native decompression: 711 MB/s, versus pigz 212 and gzip 166.
ISA-L peaks at eight workers (1158 MB/s). WASM has no ISA-L; eight workers reach 242 MB/s.
Any pipeline still emitting gzip can swap this in. Zstd, Brotli, and LZ4 often win on the speed/ratio frontier, but the installed base is gzip, ZIP, PNG, HTTP, and OCI. pigzpp's bet is to make that format as fast as the hardware allows.
Python data loaders and container builds are the obvious users: parallel gzip without leaving the process. ZIP and PNG are conveniences on the same core, not new codecs. Release artifacts statically link the engines; CI rejects dynamic zlib-ng/ISA-L deps.
The speed mostly comes from zlib-ng and ISA-L, not a new DEFLATE. The engineering is one embeddable core that five languages share. Agent-assisted modernization worked here because round-trip and cross-decode tests were strict.
All native numbers are one x86-64 WSL2 guest. No native ARM64 throughput is reported. Most comparisons use one 128 MB text corpus at one level; source, binaries, and incompressible data will look different. Language panels compare idiomatic APIs, not isolated binding overhead. CPU affinity, turbo, and frequency scaling were not pinned. "Fastest" means fastest among the implementations, inputs, and configs tested.
ISA-L's 8.5× buys speed with size. The PNG 11.8× is against Pillow's default, not every tuned encoder. The Docker number is one layer.
The AI-assisted process lives in two companion essays. The paper itself has almost no ablation of what the agents wrote, what tests missed, or how behavior diverges from pigz.