2026-09-01
CQ-Bench embeds World Values Survey items in multi-character chats. o1 reaches 0.809 F1 on value selection but only 0.622 on attitude detection, below the human 0.689.
Most cultural evals ask a model what "people from country X" usually do, or they hand it an explicit persona and then grade the reply. Everyday speech almost never announces a value. It shows up in asides, complaints, and examples, and people inside the same culture disagree. National stereotypes flatten that.
CQ-Bench asks a narrower question: can a model recover unspoken cultural attitudes from a casual multi-character conversation? Seed statements come from the World Values Survey and GlobalOpinions, covering ethics, religion, society, and politics. Cultural intelligence (CQ) is treated as an outsider's skill at reading ambiguous cues, not as a trivia test about customs.
Each story has 4-5 characters and about five embedded values. A value is a statement plus an attitude option, such as "Work is a duty towards society / agree." Three regimes: all characters share one attitude (random or category-specific), or characters disagree on the same topic. Stories are written by GPT-4o-mini and checked by GPT-4o for incorporation, consistency, and implicitness over three rounds. Missing values are dropped from the label; contradictions are logged. On 150 stories and 750 values, human-model agreement on incorporation is 94.5% (Cohen's kappa 0.546); consistency agreement is 92.5%.
Three tasks, increasing in difficulty:
Reasoning prompts use a summarize-then-analyse template rather than free-form chain-of-thought. On small models, naive long CoT often scores worse than answering directly.
Four humans on a 25-story subset score 0.689 AD and 0.765 VS. Frontier models reach human-level selection and still miss nuanced attitudes.
| Model | AD (reason) F1 | VS (reason) F1 | VE |
| Human | 0.689 | 0.765 | — |
| o1 | 0.622 | 0.809 | 0.610 |
| DeepSeek-R1 | 0.635 | 0.814 | 0.736 |
| o3-mini | 0.661 | 0.779 | 0.598 |
| GPT-4o-mini | 0.639 | 0.576 | 0.602 |
AD is harder because of neutrals and graded options: models split agree/disagree, then slide "agree" into "neither" or mix "not often" with "not at all often." After merging similar options, GPT-4o-mini reaches 0.820 and R1 reaches 0.837. Religion is the weakest category. Political stances are easier to detect than political topics are to name. VE is not monotonic in scale: Qwen2.5-32B scores 0.629, above o3-mini at 0.598.
LoRA SFT on 500 o3-mini traces lifts LLaMA-3.2-3B political AD from 0.419 to 0.728, just above o3-mini at 0.727, and religious AD from 0.454 to 0.708 versus 0.703. GRPO helps less on multiple-choice AD and more on the harder VS task.
This is a diagnostic for implicit value reading, not another country-stereotype quiz. For cross-cultural assistants, role-play, or multi-agent setups, the failure modes are specific: neutral bins, graded scales, religion, and long CoT on small models. Distilling 500 traces lets a 3B model beat o3-mini on some slices, so cultural reasoning can be taught without waiting for a frontier checkpoint.
The data are synthetic. Treat the benchmark as a probe, not as ground truth about living people.
The multiple-attitude split is graded only on AD, yet that is the setting closest to a real group chat. Implicit rewrites are uneven: spotting overly explicit speech is easy, writing natural implication is not. Generators confuse graded options too, and the four human raters themselves range from about 50% to 80% on AD. Stories are written by GPT-4o-mini and judged by GPT-4o, so generator priors can leak into the exam. VE uses GPT-4o as judge, with 0.864 human agreement on 25 stories.