Psychometric Validity Framework Proposed for Evaluating AI Benchmarks

Researchers propose applying convergent and discriminant validity from psychometrics to AI benchmark evaluation, checking whether benchmarks measuring similar concepts correlate and dissimilar ones do not, with IRT enabling item-level validity analysis.

2026-09-29 ~ 2026-09-29 · 2 related posts

• Adapting validity testing to benchmarks with IRT-based item-level analysis — sanmikoyejo · 2026-09-29
• Using convergent and discriminant validity to assess AI benchmark validity — sanmikoyejo · 2026-09-29