Synthetic benchmarks for concept bottleneck models isolate failure modes, NeurIPS 2026
berkustun · x · 2026-11-20
Researchers including Berk Ustun published "Measuring What Matters" (NeurIPS 2026 Evaluations and Datasets Track): synthetic benchmarks for concept bottleneck models (CBMs). Because few real datasets carry concept labels, CBMs are hard to train and evaluate. The benchmarks let researchers generate labeled datasets while controlling properties like data modality, concept choice, and annotation quality, and evaluate major CBM use cases—decision support and automation. Demonstrations uncover new failure modes and attribute them to architecture or data. Code is open source; install via pip install concept-benchmark.
More from Research
- DeepSeek ramps up game designer hiring, fueling game-training rumor — teortaxesTex · 2026-11-20
- ICLR 2027 desk-rejects papers after reviewers fail to file AI-use disclosure forms — algo_diver · 2026-11-20
- Chorus paper: shared n-gram tables cut memory 87% while beating DeepSeek's Engram — brown2green · 2026-11-20
- Inhabitant: training-free LLM agent completes 74/75 long-horizon household tasks — Yao Zhang · 2026-11-20
- EPFL team unveils unified 3D generative model for structure-based drug design and docking — HannesStaerk · 2026-11-20
- ICLR 2027 Review Score Distribution: 40,160 Submissions, a 3.0 Average Only Ranks Top 14% — May_F1_ · 2026-11-20