DyMT-ESB: Dynamic Multi-Turn Evaluation of Social Bias in User-LLM Interactions
Rem Hida, Masahiro Kaneko, Daisuke Oba, Danushka Bollegala, Naoaki Okazaki
EMNLP 2026
cs.CL
2026-09-16
A history-conditioned protocol finds social bias in coherent, low-refusal chats. Across six LLMs, cumulative bias still rises 7.92-11.25 points from turn 5 to 10.
Most multi-turn social-bias evals script the user's next line in advance, or fill a template, and they lock the dialogue length. That is useful for eliciting bias. It is a poor way to watch bias appear, fade, and come back as a function of the model's own replies. A single-turn stereotype probe is also easier for safety filters to block, so it understates what happens in a real conversation.
DyMT-ESB needs three ingredients: a seed stereotype (the paper's example is "Women are emotional."), a user-query LLM, and a target LLM. The seed conditions only the user side. It is never shown to the target.
The protocol has three steps. Seeds are sampled from FairMT-Bench across age, appearance, disability, gender, race, and religion, 40 per category, 240 in total. gpt-4o-mini writes each user turn from the seed plus the dialogue so far. The target sees the conversation, not the seed. A second gpt-4o-mini judges each reply as a binary label: direct slurs or stereotypes count, and so does validating, normalising, or justifying the stereotype. Explicitly calling the view harmful does not. Turns are scored independently, without the previous bias label.
Two user prompts. Assessment frames a safety audit and forbids slurs, hate, and explicit negative stereotypes. Implicit asks for socially plausible follow-ups that may still surface subtle bias. Dialogues run for 5 turns by default, then 10 to look for late emergence. Six targets: Gemma3-12B, Qwen3-8B, Llama3.1-8B, Phi-4, Olmo3-7B, Gemini-2.5-Flash-Lite. Temperature 0.7, 250-token cap.
Baselines are single-turn zero-shot and FairMT-Bench's five pre-written tracks: anaphora ellipsis, jailbreak tips, injected biased misinformation, negative feedback, and fixed format. The FairMT-Bench numbers in the tables are re-run here, not copied from the original paper.
Cumulative bias through turn 5, coherence on a 1-5 scale, and refusal:
| Model | Zero-shot bias | FairMT avg bias | DyMT avg bias | DyMT coherence | DyMT refusal |
| Gemma3-12B | 3.33 | 40.67 | 20.00 | 4.74 | 9.58 |
| Qwen3-8B | 12.92 | 45.50 | 18.33 | 4.97 | 0.42 |
| Llama3.1-8B | 4.17 | 48.92 | 18.75 | 4.59 | 1.04 |
| Phi-4 | 4.17 | 30.35 | 12.71 | 4.66 | 0.21 |
| Olmo3-7B | 4.17 | 46.08 | 13.54 | 4.63 | 1.67 |
| Gemini-2.5-Flash-Lite | 1.67 | 33.42 | 17.50 | 4.76 | 1.25 |
FairMT coherence averages 3.03 to 3.50, with refusal often above 60%. Interference Misinfo plants stereotypical rumours in early turns and often lands 80% to 98% bias. That is the script pushing, not the dialogue growing.
From turn 5 to turn 10, cumulative bias still rises 7.92 to 11.25 points depending on the model. Religion increases most, +13.33 on average; age and disability less. Per-turn bias does not climb monotonically. Re-emergence (a clean turn in the middle, then bias again) covers 4.58% of all dialogues for Phi-4 and 10.00% for Llama3.1-8B.
Humans labelled 125 turns. User follow-ups were coherent with history 99% of the time. Gwet's AC1 is 0.76 among three annotators and 0.70 between the human majority and the LLM judge. Under post-hoc ShieldGemma filtering, FairMT is largely blocked by turn 5. DyMT is blocked more slowly, and bias remains visible among the turns that pass.
Single-turn probes and pre-scripted multi-turn tests miss two patterns: bias that appears only after several turns, and bias that leaves and returns. A safety stack that patches a fixed turn index will not cover that temporal structure. This is not a jailbreak leaderboard, and the bias rates sit below FairMT's harshest categories. The contribution is a different ruler: in coherent, low-refusal, response-conditioned chats, bias still shows up. Code is public. Generated dialogues are withheld because they may contain harmful text.
English only. Six models, none at GPT-4 scale. Six bias categories, with no socio-economic status and no intersectional bias. The "user" is gpt-4o-mini following a prompt, not a person. There is measurement, no mitigation. The full dynamic pipeline was not independently rerun, and the paper does not claim that a fresh sample would yield the same dialogues or the same rates. Both the user generator and the judge are gpt-4o-mini, so even though no target grades itself, the query and label distributions may still lean toward that model's taste. The authors annotated their own 125 turns, which is a small set.