BenchHub pools 54 LLM benchmarks and 839K questions for composition-aware evaluation
tongshuangwu · x · 2026-11-23
Researchers from CMU, NYU, KAIST and others present BenchHub at NeurIPS 2026, addressing the proliferation of overlapping LLM benchmarks.
- Aggregates 54 benchmarks (MMLU, GPQA, MATH, ARC, etc.) across 10 languages — 839K questions total — reformats them into unified instances, and annotates each at the instance level with a taxonomy covering subject, skill, and cultural sensitivity.
- Exposes hidden composition/redundancy across benchmarks and enables reusable, inspectable question sets.
- Supports fine-grained, controllable model analysis and building customized evaluation sets aligned to specific applications.
More from Research
- Princeton study: diffusion models barely extrapolate basic visual skills, ~50% on counts — NielsRogge · 2026-11-23
- MIT Professor: Models Cross a Discovery Threshold Around 0.76 Log-Perplexity — ProfBuehlerMIT · 2026-11-23
- Noam Brown's 2023 bet lands: 10K agents solve Navier–Stokes, Riemann proof claimed — haider1 · 2026-11-23
- Training models to fill in omitted reasoning from proprietary model outputs — OwariDa · 2026-11-23
- NeurIPS paper dissects the Platonic Representation Hypothesis and proposes pair-free NGMA alignment — loremstasum · 2026-11-23
- Riemann proof drama: Lean community certifies it, but a geometry predicate defined as constant True raises doubts — rickasaurus · 2026-11-23