BPBench finds Chinese open-weight models dominate text compression scores
ctnzr · x · 2026-07-25
The thread discusses BPBench, a benchmark that treats a pretrained LLM as a compressor: the better the model understands language, the fewer bits it needs to encode unseen text.
It measures models on fresh human writing published after their training cutoff, using:
- a forward pass over the text
- log probabilities to compute bits-per-byte (bpb)
The author says that among open-weight models, the best compressors are mostly Chinese models, with NVIDIA’s Nemotron as the notable Western outlier in the frontier cluster. The post also notes that commercial APIs make direct compression measurement hard because they block logprobs, and sampling-based workarounds are too noisy to recover the tails of the distribution. This also makes KL/log-prob distillation against closed APIs difficult.
More from Research
- Practical multi-agent orchestration for Codex splits work into scout, worker, and coordinator roles — pvncher · 2026-07-25
- MOJO preprint mixes supervised and self-supervised losses for neural foundation models — hugo_larochelle · 2026-07-25
- AI can scan more code than humans, but engineers still own quality — ingliguori · 2026-07-25
- NVIDIA reposts a GPT-like motion model that reproduces clips with 99.98% success — Syntetisaattori · 2026-07-25
- A new AGI essay argues the field is climbing the same mountain from two slopes — op7418 · 2026-07-25
- Paper shows models implicitly optimize for hidden graders in audit-style tasks — Sauers_ · 2026-07-25