CulturalRiddles: 62-culture riddle benchmark trips frontier LLMs to as low as 22% accuracy
CohereLabsCommunity · hf · 2026-10-22
The Cohere Labs community released CulturalRiddles, a benchmark of original, human-created riddles spanning 62 cultures (45 countries, 51 languages), requiring abductive and compositional reasoning plus culturally specific knowledge.
- 28 LLM configurations (0.8B–1.6T, open and proprietary, thinking and non-thinking) were evaluated via automatic metrics and large-scale human labeling over 170,772 model answers.
- The best model reaches 79% overall, yet all models score ≤22% in some communities such as Lumasaba and Mongondow.
- Accuracy tracks language resource level: 38% high, 31% medium, 18% low, 13% very low.
- Models hallucinate rather than admit ignorance: 19% mean hallucination rate vs. only 0.7% "I don't know" answers.
All artifacts are publicly released with an open contribution process.
More from Models
- Codex reset card timing trick: burn your quota before tomorrow's expected reset — xiaohu · 2026-10-22
- Verify TTS pronunciation with Kaldi phoneme decoding: a reproducible fix for polyphonic Chinese characters — dotey · 2026-10-22
- NVIDIA's sub-1B Nemotron 3.5 Parse tops ParseBench as best open-weight doc parser — llama_index · 2026-10-22
- Alibaba ships Qwen-Audio-3.1-ASR with speaker diarization and general audio understanding — Chuanmeng Bian · 2026-10-22
- darkmux adds Flint support; dev benchmarks it against Qwen 3.6 35B on M5 Max — _rockt · 2026-10-22
- GPT-6 Astra subscription users can crank context window up to 872k tokens — JasonBotterill · 2026-10-22