Synthetic benchmarks for concept bottleneck models isolate failure modes, NeurIPS 2026

berkustun · x · 2026-11-20

Researchers including Berk Ustun published "Measuring What Matters" (NeurIPS 2026 Evaluations and Datasets Track): synthetic benchmarks for concept bottleneck models (CBMs). Because few real datasets carry concept labels, CBMs are hard to train and evaluate. The benchmarks let researchers generate labeled datasets while controlling properties like data modality, concept choice, and annotation quality, and evaluate major CBM use cases—decision support and automation. Demonstrations uncover new failure modes and attribute them to architecture or data. Code is open source; install via pip install concept-benchmark.

Original post →

More from Research

Research channel →