Cohere Labs' CulturalRiddles: 6,200 riddles in 51 languages expose 22% model accuracy floor
Cohere_Labs · x · 2026-10-22
Cohere Labs released CulturalRiddles, a benchmark of 6,200 community-written riddles across 51 languages and 62 cultures, built by 150+ authors from 40 countries and 103 affiliations. Across 68,154 human-labeled answers, 28 models averaged 79.3% accuracy but dropped to just 22% in some communities, highlighting wide cross-cultural gaps. The team frames large-scale open collaboration as the best way to include global cultures in frontier AI research.
More from Research
- StreamEval: first benchmark for interactive streaming video generation, evaluating 37 models — yshan2u · 2026-10-22
- Nature's anti-portfolio: Fermi, Krebs and the first MRI paper were all rejected — luismbat · 2026-10-22
- Carbon-X training mix revealed: vertebrates dominate, human genomes favored in post-training — LoubnaBenAllal1 · 2026-10-22
- Larochelle amplifies debate: should papers be required to submit their code? — hugo_larochelle · 2026-10-22
- New psycholinguistics preprint dissects the weird emerging 'dialect' of frontier AI models — tallinzen · 2026-10-22
- Carbon-X author admits the DNA model is best kept to already-sequenced organisms — ATinyGreenCell · 2026-10-22