One BBQ Example Pushes LLM Fairness Scores Past 90%, EMNLP Paper Questions Benchmarks
radamihalcea · x · 2026-10-27
A University of Michigan team led by Rada Mihalcea, with first author Naihao Deng, presented "One Example Is Enough to Pass Fairness Benchmarks" at EMNLP2026.
Key finding: a single BBQ example—used via one-shot learning or GRPO training—can boost LLM accuracy on fairness evaluations to over 90%.
Takeaway: passing a fairness benchmark does not imply a model is actually fair; current fairness evaluation design needs rethinking. Project page, code, and models are open-sourced.
More from Research
- Nature Physics: reprogrammable refractive index enables arbitrary multimode wave control for ML — MarioKrenn6240 · 2026-10-27
- Wakaru open-sources ontologize: labeling datasets from model activations — amplifiedamp · 2026-10-27
- EMNLP Oral paper traces multilingual representations in LLMs with cross-layer transcoders — ZhijingJin · 2026-10-27
- GPT-6 Pro reruns crime research: GTA V's 2013 launch tied to violence dips in four regions — emollick · 2026-10-27
- OpenAI's Aaron Lou on leaving a Stanford PhD for OpenAI, diffusion LMs vs autoregression — aaron_lou · 2026-10-27
- CLT-Forge: Open-source toolkit for cross-layer transcoder training and attribution graphs — ZhijingJin · 2026-10-27