OpenEvals multilingual tests: LLMs struggle with slang and dialect variants, mix locales in output
saviaga · x · 2026-11-15
Results from an OpenEvals multilingual evaluation event are in, with participant saviaga adding Spanish tests: many models really struggle with slang, since so many Spanish words have double meanings depending on context.
Key findings:
- LLMs often struggle with language and dialect variations, frequently mixing locales in the output (e.g. Spanish, BKS)
- Some specific LLMs did surprisingly well with slang, e.g. Indonesian, while others struggled
- Most LLMs, large and small, have little knowledge of spoken rural variants, e.g. Hindi
More from Models
- 4-year-old Claude account banned, user restarts from fresh account ahead of IPO — huangyun_122 · 2026-11-15
- If OpenAI's next model can't beat two-month-old Opus 5.5, it may be losing the race — OddConsideration7290 · 2026-11-15
- GergelyOrosz: Every AI Service Is Unreliable, One Subscription Isn't Enough — AccBalanced · 2026-11-15
- Gemini 4 Helix: Argon-level capabilities at 20% of the price — ThePrimeNumbers · 2026-11-15
- OpenAI now has 5 overlapping custom instruction systems; user pleads for a merge — koltregaskes · 2026-11-15
- Nex-AGI releases Nex-N2.6, a 321B multimodal agentic model built on GLM-5.3-Flash — TheRealMasonMac · 2026-11-15