ProbabilityBench: exact ground-truth uncertainty evaluation across 16 LLMs
ishidalab · hf · 2026-10-15
ProbabilityBench provides exact, instance-level class probabilities across three simulated text domains: peer review, covert information transfer, and credit risk.
- By explicitly defining each environment's full data-generating process, the benchmark analytically computes exact class probabilities and Bayes error, with a theoretical guarantee that rendering cases as natural-language memos preserves the underlying probabilities
- Exact ground truth enables direct measurement of reported probability quality via KL divergence
- Across 16 evaluated models, most on the observed cost–quality Pareto frontiers are decision models, including Jev and Perplexity Decider
- It also serves as a meta-evaluation framework for benchmarking calibration metrics and Bayes-error estimators themselves
More from Research
- kalomaze argues ensembled synthetic data can cover data-scarce domains — kalomaze · 2026-10-15
- Axiom claims proof of generalized quasi-Riemann hypothesis for all Dedekind zeta functions — basedjensen · 2026-10-15
- TorchLean expands into a verified tensor programming framework with einsum and GPU execution in Lean — burny_tech · 2026-10-15
- StreamErase hits 26 FPS real-time video object erasing on a single GPU, 200x faster than prior methods — Zhuobai Dong · 2026-10-15
- Huawei's SwarmRSI: evolving agent swarms achieve recursive self-improvement in science — Qianru Meng · 2026-10-15
- Glance Forcing cuts video distillation trajectory cost from thousands of GPU-hours to one — Zhuobai Dong · 2026-10-15