UNMASK: Discovering and Causally Verifying Spurious Shortcuts in Text Classifiers
Chidaksh Ravuru, Shashank Srivastava
COLM 2026
cs.CL, cs.LG
2026-08-10
UNMASK auto-generates boolean-expression shortcut candidates and causally verifies them via counterfactual edits, rediscovering known MNLI biases without human annotation and improving HANS accuracy by up to 12.58 pp.
Neural text classifiers often score well by exploiting surface statistical regularities in the training data rather than genuine linguistic or causal signal. In MNLI, for instance, the word "never" correlates with the contradiction label at an odds ratio of 2.86. These shortcuts prop up benchmark accuracy but collapse on adversarial or out-of-distribution inputs. The hard part is finding which shortcuts a model actually relies on without knowing in advance what to look for. Existing automated discovery methods either assume access to a bias-free reference corpus or use model errors as a proxy for group structure, assumptions that rarely hold in practice. More fundamentally, a feature being statistically correlated with a label in the dataset is not the same as the model actually depending on it; the paper calls this the gap between dataset-level correlation and model-level exploitation.
UNMASK is a four-stage pipeline requiring no human annotation. Stage 1 has a generator LLM (SCGenLLM) propose candidate surface patterns from unlabeled training examples, each returned as an executable boolean expression. That is the key design choice, since it lets every hypothesis be evaluated deterministically on the full corpus rather than staying a vague description. Candidates go through two dedup passes (first by description similarity, then by actual coverage-set overlap) plus a generate-evaluate repair loop for syntax/semantic errors, then face statistical validation: Fisher's exact test with Benjamini-Hochberg correction, requiring significance and a stable odds ratio on both a discovery split and a held-out validation split.
Statistical significance alone doesn't mean the model relies on the feature. This is where Spurious Reliance Screening comes in, checking whether the model still favors the feature's associated label even on counter-evidence examples where the true label contradicts it, filtering out features the model plausibly isn't using. Surviving candidates go through actual causal verification: a generator LLM makes a minimal edit to each feature-present example so the boolean expression flips to false while preserving the sentence's semantic relationship, verified by an independent evaluator LLM. A paired t-test on the model's predicted probability shift before and after the edit determines whether removing the feature significantly and correctly decreases the predicted probability for the associated label. Only features passing this causal test qualify. Finally, these same boolean expressions become group labels for annotation-free Deep Feature Reweighting (DFR) — no human-labeled groups needed, since the groups are literally the executable expressions themselves.
On MNLI, the pipeline finds 10 features, and causal verification confirms BERT relies on 9 of them and RoBERTa on 6, recovering the known lexical-overlap bias (entailment class) and negation/absolute-term bias (contradiction class). A notable finding is the cross-architecture split: RoBERTa is immune to three contradiction-class features (all effect sizes below 0.03, confidence intervals crossing zero), while BERT reliably exploits them. A purely dataset-level odds ratio (10.01 for both) completely misses this distinction; only the causal intervention reveals it.
| Dataset | Method | Result |
| MNLI/HANS (BERT) | PoE-IPW-Group | 64.99% (+12.58pp over ERM) |
| CivilComments-WILDS (BERT) | UNMASK+DFR | 71.84% worst-group accuracy, matching hand-labeled DFR's 70.1% |
| CivilComments discovery comparison | UNMASK vs PMI+SFV (strongest baseline) | +3.65pp worst-group accuracy |
On CivilComments-WILDS, SFV mines 10 features, six of which directly correspond to WILDS' official demographic identity tokens; causal verification confirms all 10 are jointly exploited by both architectures. Using these programmatically-generated groups for DFR gets BERT to 71.84% worst-group accuracy, nearly matching the 70.1% Kirichenko et al. (2023) achieved with hand-labeled groups, meaning no human annotation is needed to reach human-annotation-level results. As a contrast, PMI top-10 word-frequency features are entirely insult vocabulary, covering zero of the eight demographic identity axes; UNMASK covers six of eight and beats the stricter PMI+SFV baseline by 3.65 points.
The paper also runs a control experiment on sentiment classification (SST-2, IMDB) to check whether causal verification itself is trustworthy: sentiment words ARE the surface realization of the label, so they shouldn't be flagged as "logically independent yet exploited" shortcuts. Causal verification correctly returns zero confirmed features on both datasets (despite statistical significance flagging 5 and 8 high-confidence candidates), showing the causal gate actually filters out plausible-but-non-spurious correlations rather than rubber-stamping everything that passes statistics.
For teams working on model robustness or debiasing, this pipeline offers a reproducible way to locate shortcuts a model actually depends on without assuming in advance what those shortcuts look like or needing hand-labeled groups. The boolean-expression representation is what makes every discovered feature readable, testable, and directly reusable as a group label — unlike cluster-based or concept-activation-vector methods that output opaque numeric group IDs. The CivilComments result matching hand-labeled DFR is the strongest evidence this isn't just academically interesting; it's practically competitive. The RewardBench2 qualitative experiment (extending to preference data with zero task-specific modification) further suggests the methodology isn't confined to classification.
The paper's own Limitations section is direct: the entire pipeline is bounded by requiring every shortcut to be expressible as a deterministic boolean predicate. Latent, semantic shortcuts, such as topic bias, writing style, or class-imbalance artifacts, simply can't be written as boolean expressions, so the pipeline can't see them at all. That means it discovers only the subset of shortcuts that are explicitly describable, not the full set a model might rely on. Causal verification also has an inherent ceiling: a single counterfactual edit can't hold every other property of the input fixed while changing only the target feature, and when multiple features co-fire on one example, the paper removes them together, meaning the measured Δp is an upper bound on each individual feature's contribution, not a clean decomposition. The methodology also leans on two external LLMs' judgment (GPT-4o as generator, Qwen3-32B as evaluator); the paper's human evaluation against ChaosNLI shows 80.3% agreement with human majority vote, but that's a worst-case calibration on high-disagreement examples, not a direct measure of evaluator reliability across the actual screening pipeline.