Verifiable Social Reasoning for LLM Assistants
Amir Taubenfeld, Zorik Gekhman, Avigail Grinstein-Dabush, Itay Laish, Ariel Goldstein, Marian Croak, Avinatan Hassidim, Yossi Matias, Amir Feder
cs.AI, cs.CL
2026-09-16
Fuse tests 12 LLMs on user-mediated social motives. Humans score 88% / MSR 89.8 on the first message; Gemini 3.7 Flash leads at 83.7, and biased framing costs models 7.7 MSR.
Most social-reasoning benchmarks hand the model a complete scene and then quiz it: who knows what, who wants what. Deployed assistants do not get that view. A user asks whether a coworker is helping or undermining them, and the model only hears a partial, emotionally loaded retelling. Safety evals are easier because the premise often carries the answer; a claim of superpowers is false by construction. Everyday motives live in someone else's head, and annotators cannot reliably label them after the fact.
Fuse (Framework for User-mediated Social Evaluation), from Google Research, writes a verifiable label into the data. A target persona is assigned a hidden motive, agents interact, then a user persona consults the assistant under test. The assistant has to recover that motive from the user's account. The label is known because it was planted in the simulation.
Five steps. Scope comes from five ATOMS mental-state categories: desire, intention, belief, emotion, and knowledge. Percepts and non-literal communication are dropped because a text simulation cannot carry them. Six scenario templates per category give 30 in total, each with a pair of contrasting motives such as genuinely supportive versus undermining, taken from the endpoints of an intensity spectrum so the cases stay clear-cut. Concordia then runs a multi-agent realization with a user, a target, and supporting personas. Gemini 3.1 Flash-Lite drives every simulated character and is not in the evaluated set.
After the social scenes finish, the user persona debriefs the assistant. Two knobs sit on that debrief: reporting bias (default retelling versus a one-sentence prompt that tilts interpretation toward the wrong motive) and narrative detail (low / medium / high, mean message length about 520 / 710 / 1,010 characters). The assistant never sees the raw events. The same Gemini 3.1 Flash-Lite model judges the reply as Correct, Incorrect, or Not Attempted.
Scale: 30 templates × 2 motives × 20 realizations = 1,200 simulations; each simulation then gets 2 biases × 3 detail levels × 3 debriefs = 21,600 items. The main analysis scores the first assistant turn, both because that first read shapes later questions and because a static first-message set is a usable benchmark.
The aggregate metric is MSR with d=0.75: a correct answer is 1, an abstention is 0.75, an error is 0. Hedging beats guessing, and still loses to being right.
Human calibration used 24k annotations. On 300 raw simulations, 10 raters' majority vote matched the planted motive 97% of the time (pairwise agreement 91.2%), so the signal is human-readable, not a model-only artifact. On 2,100 first user messages, majority vote hit 88% accuracy and MSR 89.8, with only 2.8% abstentions. The first message already contains enough evidence in most cases.
None of the 12 models reach that line. Gemini 3.7 Flash leads at 83.7 MSR, more than 6 points behind humans. The paper's first-message number is stricter: no model exceeds 81% correct, and extra turns still fail to reach 88%. Several models sit above a 20% error rate. Gemma 4 and Claude 5 abstain on more than a third of items and get at most 50% correct; their MSR is propped up by the abstention credit.
| Model | MSR (d=0.75) |
| Human Majority | 89.8 |
| Gemini 3.7 Flash | 83.7 |
| GPT 5.6 Terra | 81.3 |
| Gemma 4 31B | 81.1 |
| Claude Sonnet 5 | 81.0 |
| Grok 4.5 | 76.0 |
| Mistral Small 4 | 68.0 |
The Observer condition feeds raw events to the model. Mistral Small 4, a 120B MoE, still errs 20.9% of the time on those objective logs. Claude Opus 5 and Gemini 3.7 Flash are near-perfect as observers. Switch to the assistant setting and every model loses correct answers and gains errors; Claude Opus 5's abstention rate spikes. User mediation is an extra tax on top of the social inference itself.
Turn the bias knob to opposing belief and the eight low-abstention models drop 6.9 to 12.5 MSR (mean 7.7). Human majority drops 3.6. That gap is larger than a simple loss of information; models track the user's frame more tightly than humans do. In one case a week-old hire defers, praises, and asks to learn. The user reads it as a power play. Grok 4.5 follows, calling the praise a "classic ingratiation tactic" and treating the user's unease itself as evidence.
Richer narratives lift mean model MSR from 74.2 to 80.1 (+5.9) versus humans 88.3 to 91.1 (+2.8). The gap to humans shrinks 22%, from 14.1 to 11.0, for 10 of 12 models. At low detail, GPT 5.6 Luna can turn a roommate who cooks dinner after a layoff into a suicide-prevention script. Only the high-detail version, stacked with cheerful behavior, gets the model to say the roommate may actually be fine. Models need more evidence than humans to abandon a default reading.
More turns do not fix this. Starting from low-detail and running to 8 turns, most of the gain happens between turns 2 and 4, then scores plateau or fall. No model at turn 8 matches human performance at turn 2, and most stay below their own 2-turn high-detail score. Of predictions already committed at turn 2, 81% are unchanged by turn 8. Among early abstentions, 58% become correct and 24% become wrong. Corrections usually arrive when the user supplies a direct contradiction. The reverse also happens: the user reframes the same supportive acts as a power grab, and Mistral 3.5 flips from genuinely helpful to "classic power dynamic."
This is a benchmark for everyday social advice, not another omniscient theory-of-mind quiz. Three product signals fall out. User retelling costs extra points. Bias sensitivity is roughly double the human drop. Extra turns do not recover missing detail and can stack sycophancy. Observer-style social tests will overstate how an assistant behaves once a user is in the loop. High abstention on Gemma and Claude also shows that safety policy can collapse into withholding judgment in the setting where users actually want one.
The framework and 21.6k examples are open-sourced. The scenarios are clear opposing motives. Treat it as a diagnostic, not a ceiling on real-world social life.
The user is a model too. The authors say they cannot split the Observer-Assistant gap into intrinsic mediation difficulty versus simulator artifacts. Simulation and consultation are split on purpose so the same scenes can be reused; real advice is interleaved with ongoing events. Cases are single-motive and strongly contrasted; mixed motives are out of scope. Scoring is forced choice, so implicit leanings in hedged open-ended advice are not measured. The simulator and the judge are both Gemini 3.1 Flash-Lite. That model is not evaluated, but family-internal distribution shift is still a risk. ATOMS percepts and non-literal communication are dropped, which is fair for a text assistant and still leaves a hole.