tufazip: Transformer-Based Lossless Compression Squeezes 1GB Wikipedia to 105MB
aigclink · x · 2026-11-20
tufazip applies a Transformer to lossless compression (truly byte-exact, not regeneration), shrinking 1GB of Wikipedia text to 105MB — an 89.46% reduction on enwik9 (100.96MB compressed + 4.43MB decompressor). It works by using the model's next-token probability distribution with arithmetic coding. The catch: it took two NVIDIA B200 GPUs about 7.3 days to compress and 7.5 days to decompress, excluding offline training — so it's nowhere near practical yet.
More from Research
- François Fleuret asks: is the gradient Japanese? It never refuses to give you a direction — francoisfleuret · 2026-11-22
- Hat Matrix Explained: How Much Leverage Does Each Data Point Have in Least Squares? — RobertTLange · 2026-11-22
- Energy-Based Reasoners: gradient descent on latent states for adaptive reasoning — kalyan_kpl · 2026-11-22
- Lean community certifies Riemann proof; honeypot fake keys expose breaches at three AI providers — LeviTurk · 2026-11-22
- Riemann hypothesis 'proven by Turkish mathematician using AI' claim mocked as the 'Organic Turk' — CatAstro_Piyush · 2026-11-22
- Andriy Burkov releases The Hundred-Page Language Models Book, hands-on with PyTorch — burkov · 2026-11-22