CMU and Bosch's BestRand cuts benchmark compression high-error risk by 50.2%
CarnegieMellonU · hf · 2026-11-06
A CMU/Bosch Research paper introduces BestRand for stable, low-cost benchmark compression. Key points:
- Derived an analytical approximation of the error distribution when estimating full-benchmark scores from subsets: errors are long-tailed — random sampling is strong with high probability but unstable due to rare large errors.
- This instability is inherent to modern foundation models and persists as more models are evaluated.
- BestRand keeps random sampling's efficiency while substantially shrinking the high-error tail.
- Validated on 70+ text, multimodal, and agent benchmarks, extended to VLA evaluation; reduces high-error risk by 50.2% and estimation error by 16.4% at only 51.3% of the construction cost, and cuts OOD error by 21.8% across shifts.
- Code and data are public on GitHub.
More from Research
- UBC and Sony release Solv3R, turning a looped Transformer into a reusable 3D geometry solver — CSProfKGD · 2026-11-06
- Discovery Loop member launches $DISCODEV open AI materials-discovery loop — arkitus · 2026-11-06
- Northwestern turns ordinary smartwatches into hand and gesture sensors with TransfHAR and electrode-based system — realkaranahuja · 2026-11-06
- Three researchers solo-prompting Bing Chat for four weeks gets roasted as a study — Sauers_ · 2026-11-06
- New attention kernels meet real models: where interpretability overlaps systems research — QuentinAnthon15 · 2026-11-06
- MoE All-to-All Comms Eat 60% of Training Steps; Correlated Placement Cuts 58% of Dispatched Rows — QuentinAnthon15 · 2026-11-06