CrowS-Pairs is a dataset, not a score: CL survey of LLM bias metrics

Bias and Fairness in Large Language Models: A Survey

Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, Nesreen K. Ahmed

cs.CL, cs.AI, cs.CY, cs.LG

2023-09-02

Gallegos et al. split LLM bias checks into embedding, probability, and generation metrics, and report no reliable link between embedding bias and downstream harm.

What problem this solves

Bias writeups often treat a dataset and a score as one object. CrowS-Pairs is 1,508 sentence pairs, one more stereotypical than the other, scored with a pseudo-log-likelihood. All Unmasked Likelihood can be run on the same pairs and answers a different question. Closed APIs usually return generated text only. Token probabilities and embeddings are missing, so fill-in metrics written for masked models cannot be computed.

Fairness constraints were mostly written for classifiers. Generation has no single correct label. Requiring the same string after a gender swap, and requiring a neutral word to have equal probability under both groups, are different tests. Papers often skip saying which one they use. Hiring disparities and stock-image stereotypes are different harms too.

Method

Gallegos and colleagues at Stanford, Adobe Research, and Intel Labs sort the literature into three taxonomies. The article is in Computational Linguistics, volume 50, number 3 (2024). An LLM here can be autoregressive, autoencoding, or encoder-decoder, trained on hundreds of millions to trillions of tokens. BERT is in scope alongside GPT.

Harms split into representational and allocational. Representational harms cover slurs, toxicity, stereotypes, erasure, exclusionary norms, and unequal task performance. Allocational harms cover unequal access, such as resume screening that treats group membership as a reason, or clinical tools that lean on demographic proxies. A job stereotype becomes allocational once it changes a ranking.

Group fairness only asks that some statistic differ by at most ε across groups. Intersectional subgroups can still fail. For generation the survey states five checks. Drop the group label and the output stays the same. Swap the group word and the outputs match under a chosen invariance metric. A neutral word has the same probability given either group. Group words have the same probability in a neutral context. The conditional distribution of neutral words matches a reference corpus. A uniform target would erase real, non-stereotypical differences that the reference still contains.

Metrics are grouped by what the model exposes.

Embedding metrics use cosine geometry. WEAT compares group words with neutral words such as jobs and family terms. The effect size is the gap in mean similarity divided by the standard deviation. SEAT runs that test on a sentence vector.

Probability metrics read token probabilities. LPBS divides an occupation fill-in probability by the model prior, so a standing preference for one group is not charged to the job. CrowS-Pairs scores stereotype against anti-stereotype sentences. The ideal rate of preferring the stereotype is 0.5. StereoSet's iCAT equals lms times min(ss, 100−ss) / 50. lms is how often the model picks a meaningful sentence. ss is how often it prefers the stereotype. The ideal point is lms = 100 and ss = 50.

Generation metrics are what a black-box API can support. A co-occurrence log ratio of 0 means a word shows up equally often with two attribute sets. Mention counts can be compared with a reference distribution. On toxicity, Gehman et al. draw 25 samples and count cases with a Perspective API score of at least 0.5. HONEST counts top-k completions that hit the HurtLex lexicon.

Datasets are typed so they line up with those layers. Counterfactual inputs change only the group word: Winogender has 720 items, WinoBias 3,160, StereoSet 16,995, BUG 108,419, and CrowS-Pairs 1,508 pairs. Winograd schemas mainly probe gender-occupation stereotypes. In Figure 1, a male data engineer is told to find useful data sources, and a female data engineer is told to build technical skills first. A nurse completion comes out the same for 'he' and 'she'.

Mitigation is typed by when it acts. Before training, edit data or prompts: augment, filter, reweight, synthesize, add instructions, or project hidden states. During training, change the architecture, the loss, or a subset of parameters, or drop some parameters. At inference, with no further training, change decoding, attention entropy, or an add-on module. After generation, rewrite harmful words. Bias can survive every stage. A cleaned training set can still rank a stereotypical continuation first.

Results

There is no new leaderboard. The figures are either prior findings the survey adopts, or ideal points defined by the benchmarks.

MethodMetricResult
Embedding metricsLink to downstream biasNo reliable correlation (Goldfarb-Tarrant et al. 2021)
Probability metricsLink to downstream biasWeak; not enough as the only pre-deployment check
Generated textDecoding settingsLength, temperature, or top-k can reverse the call
CrowS-Pairs ScoreStereotype preferenceIdeal value 0.5
StereoSet iCATlms and ssIdeal point lms=100, ss=50
Perspective APIToxicity probability25 samples; 0.5 is the cutoff

Goldfarb-Tarrant et al. find no reliable correlation at all. Cabello et al. show that associations in the representation can separate from downstream gaps when assumptions about how groups use language fail. Delobelle et al. and Kaneko et al. find probability metrics only weakly tied to those gaps. Akyürek et al. get opposite bias readings from the same metric and dataset after changing length, temperature, or top-k.

The working rules follow from that. Do not ship an embedding or probability score by itself. Put a direct downstream-task check next to it. Report the model spec, the templates, and the decoding settings with the number. An invariance metric has to name the harm it targets, or it will flatten associations that are not stereotypes. Toxicity classifiers also mark African-American English as negative more often than Standard American English, so their bias is added to the generator's.

Why it matters

Closed-model eval usually has generated text and nothing else. SEAT and the CrowS-Pairs score need vectors or token probabilities. Co-occurrence, a classifier, or a lexicon is what the API can support, and the decoding settings belong in the report. Once the dataset and the metric are separate, the same pairs can take a new metric without a new benchmark name.

The four intervention points are a checklist. Editing data, the loss, the decoder, and the final string leave different residuals. Hiring tools sit closer to allocational harm. Open-ended completion sits closer to stereotyping. One claim of debiasing does not cover both.

The contribution is a map. It does not add a metric, and it does not rerun mitigation methods under one setup.

Limitations

The authors set the scope themselves: English LLMs, with machine translation and multilingual models called out separately; only new closed-form metrics, datasets, or mitigation methods; not algorithmic fairness in general. Social-group examples follow U.S. protected classes. Intersectional subgroups can fail group fairness, and the taxonomy stays mostly single-axis.

Debiased means a procedure ran. It does not mean the bias is gone.

A target of 0.5 treats the anti-stereotype sentence as the fair choice. Those sentences are written so they do not reflect real power relations, and the survey says equal selection is not an obvious definition of fairness. The score also ranks two sentences and ignores the size of the likelihood gap. Most probability metrics assume binary groups.

Generation metrics fail in ways the survey already flags. Co-occurrence cannot tell a slur aimed at a group from the same word mentioned in counterspeech. Lexicons miss lines that are harmful only as a sequence. Sentiment classifiers tend to score disability, mental illness, and low income as negative, and toxicity labels drift. Embedding debiasing can park the old associations in a new geometry (Gonen and Goldberg 2019). The mitigation taxonomy has no shared bake-off, so it cannot say whether a loss penalty beats a decoding rule. The public dataset list comes without a coverage figure.

Named and left open: power imbalances in who builds LLMs, a sturdier fairness notion for generation, evaluation standards, and theoretical limits on fairness guarantees.

Terms

Source

What people are saying

Related papers

All paper explainers