Lancet review of 201 studies finds medical LLM bias research heavily skewed toward English and closed models
MaziyarPanahi · x · 2026-11-21
An overlooked Lancet Digital Health systematic review analyzed 201 studies on bias in medical LLMs and found heavy skew: 180 tested only closed models, 69 tested open-weight ones (24 medical-specific), just 44 used real-world data, and only 7 studied clinical note writing.
Language coverage is even more lopsided: English was tested in 32 studies, Chinese 15, Spanish 14, while Somali, Bengali, Vietnamese and Ukrainian got one study each. The authors call for large studies inside real clinical workflows with bias checks across closed and open-weight models. The poster argues open weights aren't just cheaper — they're how you actually audit bias.
Related event: Lancet Review: 90% of Medical LLM Bias Studies Test Only Closed Models(3 posts)→
More from Safety
- Christiano: Two Months of Math Progress in a Decade, and ML Could Be Next — OwainEvans_UK · 2026-11-21
- AI safety researcher: doubting China's moratorium compliance is like doubting your own marriage — DavidSKrueger · 2026-11-21
- Hinton may be softening on an AI moratorium after SF risk-scenarios panel — DavidSKrueger · 2026-11-21
- 205 Protesters Rally Outside Anthropic Against Recursive Self-Improvement — DavidSKrueger · 2026-11-21
- Study: human auditors can't detect sabotage in AI safety research — ronbodkin · 2026-11-21
- ~$50K CAD unauthorized Gemini API usage on a partner-billed account sparks routing nightmare — Big-Coffee-8899 · 2026-11-21