Columbia EMNLP paper: dense retrievers are systematically biased by identity signals in queries
vishalmisra · x · 2026-10-28
A Columbia University team presented at EMNLP2026 that dense (embedding-based) retrievers — the core of many RAG systems — are systematically skewed by identity signals in queries. Key findings from the poster: retrievers favor the asker's ideology in both synthetic and real queries; they perform significantly worse on Asian-American health queries, with gaps of 17+ percentage points on real queries, larger than in synthetic tests; and lexical control (hybrid BM25) only partially mitigates bias, suggesting deeper representation-level causes beyond token features. Evaluated across politics/religion/health domains with synthetic and naturalistic experiments.
More from Safety
- European 'sovereign AI' startup Giotto bans benchmarking in its ToS, drawing flak — teortaxesTex · 2026-10-28
- p(doom) became a popular term — how seriously should we take the risks? — Unfair-Journalist-66 · 2026-10-28
- Why AI cyber capabilities surged since 2022 but cyber damages didn't: Joshua Saxe analysis — joshua_saxe · 2026-10-28
- EU governments approve GDPR change letting AI train on personal data without consent — IgorCarron · 2026-10-28
- Cyphrex Launches MCP Gateway to Block AI Agent Actions with Audit Trail — modelcontextprotocol · 2026-10-28
- I put an AI in the Garden of Eden — the serpent's prompt injections haven't worked yet — bios444 · 2026-10-28