BPBench finds Chinese open-weight models dominate text compression scores
ctnzr · x · 2026-07-25
The thread discusses BPBench, a benchmark that treats a pretrained LLM as a compressor: the better the model understands language, the fewer bits it needs to encode unseen text.
It measures models on fresh human writing published after their training cutoff, using:
- a forward pass over the text
- log probabilities to compute bits-per-byte (bpb)
The author says that among open-weight models, the best compressors are mostly Chinese models, with NVIDIA’s Nemotron as the notable Western outlier in the frontier cluster. The post also notes that commercial APIs make direct compression measurement hard because they block logprobs, and sampling-based workarounds are too noisy to recover the tails of the distribution. This also makes KL/log-prob distillation against closed APIs difficult.
More from Research
- Tencent's HiSTEVE 1.0: hierarchical VLA agent plays Minecraft with native keyboard and mouse, hits 85% SOTA — _akhaliq · 2026-10-26
- SJTU's VaM lets robots learn online from mistakes, 95%+ success over 4-hour autonomous runs — chris_j_paxton · 2026-10-26
- Hugging Face uses small open models to reverse engineer a game binary byte-for-byte — yb2698 · 2026-10-26
- Author hopes contracting formalism becomes AI's "MDP moment" for agent economics — xuanalogue · 2026-10-26
- Frontier LLMs negotiate contracts poorly and frequently renege for selfish gain, study finds — xuanalogue · 2026-10-26
- ContractSim: first framework evaluating LLM negotiation over contingent natural-language contracts — xuanalogue · 2026-10-26