TukaBench at EMNLP: African-language prompts sharply cut LLM refusals
davlanade · x · 2026-10-26
Researchers presented TukaBench at EMNLP 2026 in Budapest: a culturally grounded jailbreak benchmark for 7 African languages extending JailbreakBench via four settings (human translation, culturally adapted translation, human-curated prompts filtered via GPT-5.2 interaction, and code-switched prompts).
Key findings:
- Across closed and open models, prompting in African languages reduces refusal rates vs English; culturally adapted prompts yield the least refusal.
- Two structural issues surfaced: comprehension failures in low-resource languages (addressed via a new "Deflection" label) and degraded LLM-as-a-judge reliability, with judge-human agreement dropping for lower-resource languages and scripts.
More from Safety
- Prompt turns Claude Code into an iPhone backup forensics tool exposing 30-day 'deleted' photo copies — alex_verem · 2026-10-27
- Anthropic reveals it bans customers worth 5-7% of revenue every 9 months — Polymarket · 2026-10-27
- AVERI independently audits Amazon's internal AI safety evals in landmark pilot — Miles_Brundage · 2026-10-27
- US chases AGI dominance while China, EU and ASEAN treat AI as public infrastructure — BubblyExtent1693 · 2026-10-27
- Agent Field Kit launches $29 security runbooks for production AI agents — alvelda · 2026-10-27
- DeepSeek Safety Eval Finds "Over-Attribution": Model Mistakes Ordinary Tasks for Tests — Much_Boysenberry9124 · 2026-10-27