CulturalRiddles: A Multicultural Benchmark of Riddles
My Chiffon Nguyen, Tianyi Hu, Vinod Anbalagan, Henry Gagnier, Pouya Sadeghi, Farah Abdou, Tidor-Vlad Pricope, Diego MMS Oliveira, Jocelyn K D'Arcy, Madeline Smith, Oluwaseun A Ajayi, Yusser Al Ghussin, Zena Al-Khalili, Micol Altomare, Shayan Amir, Ahanaf Aziz, Bryan Chen, Ruchira Dhar, Abdellah El Mekki, Rawan Mohammed Elframawy, Akhil Eppa, Silvia Fernandez, Alex Gimei, Arya Hariharan, Christian Hoang, RamMohanRao Kadiyala, Diana Kozachek, Peerat Limkonchotiwat, Ankita Maity, Di Mo, Ayush Munot, Aylin Naebzadeh, Anastasiia Nikitina, Dina Pisarevskaya, Divas Poudel, Jebish Purbey, Md Rabius Sany Apu, Ahmad Mufli Ramadhan, Ralitza Nikolova Rizova, Diogo M. Serrano-Miranda, Vivek Gopalkrishna Silimkhan, Luis Fernando Solis Navarro, Kaiser Sun, Arya Suneesh, Manaswitha Sunkara, Tejaswi Sunkara, Jonathan E Swindell, Hanna Yukhymenko, Jingyi Zheng, Esther Adenuga, Ilia Afanasev, Wafa Aissa, Lisa Alazraki, Lauren C Altomare, Priscilla Amondi Amuok, Emile Anand, Gabriel Assis, José Matías Avalos, Silvia Lorena Avila, Zakaria Baannou, Niyati Bafna, Yash Bagla, Sarah Bahrami, Bontu F Balcha, Vanya Bannihatti Kumar, Irene Baucells, Abdumutalib Begmuratov, Naman Bhatia, Souvik Bhattacharyya, Isabel Blanco Ferreiro, Felipe Bochini, Sayuru Bopitiya, Angana Borah, Tulika Bose, Olena Bugaiova, Juliana Carvalho, Nancy F Chen, Stephy Chen, Rochelle Choenni, Jayanth Krishna Chundru, Leandro A Cossio, Nebojša Damnjanović, Ona de Gibert, Marco De Santis, Pedro Delgado, Kaustubh Dholé, Adnan El Assadi, Wesam El-Sayed, Nadine M Elframawy, Oumayma Essarhi, Naome A Etori, Reuben Chagas Fernandes, Saanidhi Pravin Gade, Javier García Gilabert, Álvaro González-Escapa, Bin Hu, Kaiyu Huang, Abderrahmane Issam, Danka Jokić, Ulugbek Khalilov, Dominik Krzemiński, Roy Ka-Wei Lee, Haichang Li, Chen Cecilia Liu, Zhengyuan Liu, Cedric Lothritz, Houman Mehrafarin, Antonio Miranda-Escalada, Fengran Mo, Mehrnaz Mofakhami, Wafaa Mohammed, Felipe Moreno-Vera, Prahitha Movva, Jannatul Nayem, Van Ngo, Joy Olusanya, Marc Pàmies, Avinash Kuney Pandey, Yurii Paniv, Fred Philippy, Vladislav Poritski, Dikshya Poudel, Joanna Radoła, Sara Rajaee, Swati Rajwal, Prerana Rane, Muhammad Ravi Shulthan Habibi, Lucas Resck, Shubhashis Roy Dipta, Ananya Sahu, Eduardo Sánchez, Borja Sánchez Clemente, Aleix Sant, Nobin Sarwar, Istiak Shihab, Santhoshini Pranathi Singaraju, Jane Smith, David Stap, Emily Z Su, Ananth K Suresh, Sajag Swami, Ruiyang Tan, Nazia Tasnim, Khalid Tounoussi, Patteera Triamamornwooth, Muhammad Saad Umar, Shankar Venkitachalam, Oksana Volchek, Lingbo Wang, Debela Desalegn Yadeta, Rushikesh Zawar, Jinghan Zhang, Mike Zhang, Weihua Zheng, Marzieh Fadaee, Akhil Arora, Károly Boczka, Julia Kreutzer
cs.CL, cs.AI
2026-10-22
167 native speakers wrote 6,191 riddles across 62 communities. Of 28 LLMs, the best reaches 79.4% overall, but none exceeds 22% on Lumasaba or Mongondow.
Cultural benchmarks mostly check whether a model remembers a fact. BLEnD, CulturalBench, and Global-MMLU are built around multiple choice or recall, coverage is uneven, and human grading often stops at a few hundred answers in fewer than ten languages. Asking what 9 November is called in Germany only tests whether the name Schicksalstag is stored. The riddle about the German 9/11 first requires reading the date in day-month order, then linking 9 November to the name Germans actually use. The clues have to be combined, and the right answer is the one that best explains all of them together. That is compositional reasoning plus abductive reasoning, along with associations that hold inside one community.
Earlier cultural-riddle sets cover one to three languages. On MultiNRC, no model’s judge score exceeds 50%. BanglaRiddleEval tops out near 56% in multiple choice, against 83% for humans. CulturalRiddles scales the task to 6,191 newly written open-ended riddles from 62 communities.
The unit of collection is a national-linguistic community still in use, such as Hungarian in Hungary or Spanish in Mexico. Cohere Labs Community recruited 167 volunteers who, over about six months, wrote 6,191 original riddles for 62 communities, 51 languages, and 45 countries. The target was 100 riddles per community: ten on each of eight shared topics, plus ten on each of two topics the community chose. Food is 27.4% of those 124 wildcard topics.
Communities wrote from scratch and did not translate one another. The questions are indirect, and reference answers are usually one to four words, built from local nicknames and jargon. A second native speaker checks that each riddle is solvable, the clues are unambiguous, and the reference is right. Not every member of the community is expected to solve it. In a survey of 76 respondents from 43 communities, 97.4% said they drew mainly on personal experience. Seven languages appear in more than one country, seven countries contribute more than one language (ten in India), and Arabic is included in five varieties. Buginese, Mongondow, Masaaba, Konkani, and Sudanese Arabic were absent from earlier cultural benchmarks.
Twenty-eight model configurations received the same English prompt, with country and topic filled in. Each answered 6,090 riddles, 170,520 responses in total. One hundred Spanish (Spain) riddles arrived too late for the runs. Sizes run from 0.8B to 1.6T parameters. One GPT-5.5 configuration had web search on.
Scoring takes three routes. Rules cover 18.87% of responses: a case-insensitive exact match counts as correct, empty outputs and copied placeholders count as wrong, and an explicit I-don’t-know is stored as its own score. Native speakers labeled 68,067 distinct answers, covering 100,769 responses (59.10%). Gemini 3.5 Flash and DeepSeek V4.1 Flash scored the rest with model names hidden. A score is kept only when the two judges agree. The 1.78% they split on are left out of accuracy. Accuracy counts fully correct answers only. Partial credit and abstention do not count as correct. Model comparisons use the 57 communities with valid human evaluation.
On items labeled twice, native speakers agree at Cohen’s κ of 0.80 on correct versus incorrect, and at quadratic-weighted κ of 0.80 on the 0-3 score. Judges agree with humans at about 0.79 to 0.84. Agreement on 13 error types is only 0.41, so those types are descriptive. The judges are soft on the strongest models: together they accept 9.0% of the answers humans rejected for Gemini 3.1 Pro, and 6.9% for GPT-5.5. On answers that are valid under a different wording, a single judge rejects 25% to 29%.
| Model | Overall accuracy | Correct answers missed by exact match | Communities under 50% |
| Gemini 3.1 Pro Preview | 79.4% | 39% | 6/57 |
| GPT-5.5 (default) | 76.3% | 44% | 9/57 |
| Claude Opus 4.7 Thinking | 66.0% | 48% | 9/57 |
| GPT-5.5 (search) | 64.5% | 41% | 14/57 |
| DeepSeek V4 Pro (high reasoning) | 63.8% | 43% | 11/57 |
| Gemini 2.5 Flash | 50.1% | 50% | 27/57 |
| Gemma 4 31B | 39.6% | 48% | 35/57 |
| Qwen3.5 0.8B | 0.4% | 68% | 57/57 |
Two configurations still above 50% are left out of the table: Claude Opus 4.7 without thinking at 59.7%, and DeepSeek V4 Pro at its default setting at 58.8%. Every model ranked below Gemini 2.5 Flash scores under 40%. At 8B parameters or fewer, the ceiling is 10.4%. Gemma 4 31B at 39.6% beats OLMo-3 32B Think at 11.3% and both Apertus 70B models, at 19.6% and 15.2%. The best open-weight system, DeepSeek V4 Pro with high reasoning, ranks fifth.
Thinking helps on the pairs that were tested. Turning it on lifts Claude Opus 4.7 by 6.3 points, and high reasoning adds 5.0 points for DeepSeek V4 Pro. Web search drops GPT-5.5 from 76.3% to 64.5%. Among those new errors that received a human type, 39% name the wrong entity in the right category and 25% are made up. A Swahili riddle asks for the home of the Lunatic Express. Several models return the Uganda Railway and miss the Railway Museum in Nairobi.
Exact match misses about 47% of answers later marked correct. After human and jury review, accuracy is 64% to 166% higher than exact match for every model except Qwen3.5 0.8B. Claude Opus 4.7 Thinking moves from fifth to third. Among 132,940 responses scored by rules or humans, hallucinations are 24.19% and admissions of not knowing are 0.80%. The prompt allows abstention. The highest abstention rate of any model is still 3.81%.
The eight easiest communities average 47% to 66% across models, and the median model reaches 73% on English in the United States. The eight hardest average 4% to 14%. Gemini 3.1 Pro and GPT-5.5 average 43% and 39% there, about half their overall scores. No model exceeds 22% on Lumasaba or Mongondow.
Pooled accuracy falls with Common Crawl page counts: 37.0% for high-resource languages, 30.8% for medium, 18.0% for low, and 12.7% for extremely low. Inside a band, the standard deviation across communities is still 8 to 13 points. Swahili is medium-resource, and Sudanese Arabic inherits Arabic’s high-resource label, yet both sit in the hardest group. Simplified Chinese in China averages 28.5%, against 46.3% for Chinese in Singapore. Five models then answered English translations from five of the hardest communities. On 2,049 responses scored in both conditions, accuracy moves from 22.1% to 32.4% (plus 10.3 points, 95% interval 7.6 to 13.0). The same models reach about 69% on high-resource communities, so translation closes about a fifth of the gap. Sudanese Arabic shows no detectable gain. Buginese, Lumasaba, and Yoruba gain 14 to 18 points.
Regional Tiny Aya variants show no significant difference inside their target regions (Cochran’s Q, all p greater than 0.48). Qwen-SEA-LION-4.5 27B is 5.5 points above Qwen3.5 27B overall, with no extra gain in its target regions. On the benchmark side of a crossed random-effects model, individual riddles account for about 60% of the variance, language 24%, community within a language 13%, and topic 2%. Model identity accounts for 38% of the total. About a third of riddles use a very rare word in the English reference, and their adjusted odds of a correct answer are about half those of common answers (OR 0.54, interval 0.46 to 0.65). A Konkani riddle asks for bebinca, a layered Goan sweet. GPT-5.5 and Claude answer kuswar, the Christmas platter that contains it.
A new frontier model can be scored first with rules and with human labels already collected on other models. That covers about 80% of a leading model’s answers, leaving roughly 100 to 115 for a native speaker. Coverage falls to 51% for the median model and to 2% to 13% for the smallest models, whose wrong answers rarely repeat.
After adjustment, the odds of a correct answer on low-resource and extremely low-resource languages are 0.10 (interval 0.02 to 0.48) and 0.04 (0.01 to 0.15) times the odds on high-resource languages. The point estimates are about a 10-fold and 25-fold gap, and the intervals are wide. Raw accuracy is milder: 37.0%, 18.0%, and 12.7%. English translation closes about a fifth of the hardest communities’ gap to high-resource ones, communities that share a language can differ by 17.8 points, and regional adaptation adds no extra accuracy in the regions it targets. The missing piece is local names, allusions, and ways of remembering. A final right-or-wrong score cannot separate a missing fact from a misread clue.
Exact match changes rank. String hits alone inflate the gaps between models on open cultural answers.
Most communities had one or two writers. A second native speaker checked solvability and the reference answer, so the cultural coverage is still whatever those writers chose to ask. About a third of surveyed writers said the riddles depend on widely known knowledge, and 26% said the knowledge is mostly local. Whether another group of writers would name the same hardest communities was not tested.
The abstract describes the riddles as created entirely by humans. In the survey, 46.1% tested difficulty with AI tools, 39.5% used AI to brainstorm, and 5.3% used AI to paraphrase or co-write. The comparison table marks human-only authorship as partial. Whether any of that text entered the logs of the models under test was not checked.
About 22% of answers are scored only by the jury. The judges accept some human-rejected answers from the strongest models and reject 25% to 29% of differently worded correct answers, so the biases pull in both directions. The paper does not convert that into a net shift in overall accuracy, which means 79.4% and 76.3% should be read with slack. Error types are too noisy for fine rankings; 24.19% is a magnitude. The translation study had no native-speaker review and dropped 41 translations that contained the answer, 38 of them from Lumasaba, so the one-fifth estimate is less stable for Masaaba.
The main prompt is English throughout, with no arm that instructs the model in the local language. The 17.8-point gap between Chinese in China and Chinese in Singapore is not split into training coverage, which riddles were sampled, and writer style. All Sindhi (India) human labels were invalid, and three further communities had none, which is why the reported comparison covers 57 communities.