BenchMIRT audits LLM evals: BBQ questions test reasoning, not safety
allen_ai · x · 2026-09-02
The Allen AI (Ai2) team released BenchMIRT, a tool using psychometrics-inspired multi-dimensional IRT models to audit LLM safety and capability benchmarks, checking which model abilities their questions actually test.
One key finding: on BBQ, a social-bias eval, BenchMIRT found the questions distinguished models more by reasoning ability than safety — suggesting the safety-capability tradeoff may partly be an artefact of how benchmarks measure safety.
Related event: Ai2's BenchMIRT Audits What LLM Benchmarks Really Measure(4 posts)→
More from Research
- TRACE Open-Sourced: Migrating G1 Controller to Full-Size Humanoid H2 in 8 Hours on One GPU — zhengyiluo · 2026-10-18
- Humanoid Robot Funding Tops $3B as Brian Potter Explains the VLA Models Driving Robots — scottleibrand · 2026-10-18
- Terence Tao on 'War Footing' as OpenAI Claims Navier-Stokes Solve Shakes Mathematics — AlexTensor · 2026-10-18
- Univ. of Tokyo paper: why your recorded voice feels eerie — a vocal uncanny valley — ssh4net · 2026-10-18
- MIT's SANDO Drone Planner Avoids Every Moving Obstacle Across 12 Real Flights — Olivier__OG · 2026-10-18
- No-backprop pretraining speedup of 2.3x claimed, inspired by mech interp essay — CatAstro_Piyush · 2026-10-18