Ai2 Releases BenchMIRT to Audit Benchmarks, Reveals BBQ Tests Reasoning Not Bias
allen_ai · x · 2026-09-02
Ai2 released BenchMIRT, a new method based on Item Response Theory (IRT) to audit LLM benchmarks at the prompt level. It found that BBQ, a social-bias eval, actually distinguishes models more by reasoning ability than safety. The tool helps separate signals and clarify what drives benchmark scores.
Related event: Ai2's BenchMIRT Audits What LLM Benchmarks Really Measure(4 posts)→
More from Research
- Stanford and Together Propose Next-Concept Modeling, a Third Paradigm for Text Generation — TheTuringPost · 2026-10-18
- Opus-assisted open design: a $31k low-field head MRI sketched in hours — DreamFly_13 · 2026-10-18
- semiconductor.diy: open-source guides to make real chips in a garden shed — jwt0625 · 2026-10-18
- Work in progress: geometric structure of GPG projections as Markov chain coarsening — michaelchchoi · 2026-10-18
- Backprop-free pretraining gets 2.3x speedup as Noise method crushes Dust record on 8 GPUs — inductionheads · 2026-10-18
- q3dviewer: open-source Qt+OpenGL tool for painter-like point cloud editing — 4310sy · 2026-10-18