500K Names Tested: Unequal Tokenizer Support Skews LLM Judgments in Hiring and Lending

Mir Tafseer Nayeem · hf · 2026-09-30

A new paper argues that fairness evaluations using "matched names" fail at the lexical interface: matched names are not necessarily matched inputs.

Some names get direct single-token access while others are assembled from multiple subwords, creating unequal name-surface support.
Across nearly half a million first names and 12 LLM-associated tokenizers, direct lexical access is highly selective, model dependent, and uneven across race- and gender-associated name metadata.
The authors introduce NameTrace, a model-native, fine-grained, pre-behavioral framework measuring whether unequal name-surface support remains a vocabulary property or becomes visible in task-relevant internal representations.
On matched atomic and short-fragmented names within the same race/ethnicity-gender strata, support predicts systematic differences in concept accessibility across fellowship, hiring, clinical assessment, and lending; differences persist across all eight matched strata, extend across model families, and transfer to unseen names.
Hidden-state interventions show the measured task directions have downstream leverage, shifting later constrained choices.

The conclusion: unequal lexical support is demographically structured at the input and remains visible in task-relevant model computation; behavioral comparability begins with lexical comparability.

Original post →

500K Names Tested: Unequal Tokenizer Support Skews LLM Judgments in Hiring and Lending

More from Safety