Zyphra open-sources PUFFER, a CPU-based deduplication system 35x faster
bclavie · x · 2026-09-02
Zyphra Research released PUFFER, a novel incremental fuzzy deduplication system for LLM-scale datasets. Running entirely on CPU, PUFFER achieves an 11-35x speedup over existing methods. It is used internally by Zyphra and released under the Apache 2.0 license.
More from Infra
- Is it worth switching to AMD for AI Video Generation? ROCm Inquiry — iridescentblob · 2026-09-02
- HybridInfer: Router auto-falls back to cloud when local model wedges — simrankoulsm · 2026-09-02
- Looped transformer rumors may be true, bearish for HBM demand — zephyr_z9 · 2026-09-02
- Buying GPUs to run local models is pointless, argue devs: cloud APIs win on cost — brandon_galang · 2026-09-02
- 4,000 GB200s arrive in Texas for Horizon TACC, billed as largest academic supercomputer — AlexGDimakis · 2026-09-02
- Huawei's $18B R&D spending rivals Nvidia, while profit slumps to $3.5B — teortaxesTex · 2026-09-02