Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learning
Jiayi Yuan, Hangoo Kang, James Jihao Liu, Yejin Choi, Vikram Iyer, Liwei Jiang, Natasha Jaques
cs.CL, cs.AI, cs.LG, cs.MA
2026-09-14
MoDA is online RL that conditions a shared LLM on numbered roles; Infinite-Chat SBERT diversity rises 265% and average pass@1 rises 10.3 points over Qwen3-8B.
Alignment training packs an LLM onto a handful of high-reward replies. Diversity falls even when many other answers are equally valid. Scientific ideation and creative writing need a spread of plausible outputs, not one canonical paragraph with all the probability mass.
Inference-time prompting and decoding can paper over the collapse without changing the training objective. Training-time diversity rewards get hacked by language switching, padding, and off-topic text. The missing piece is a method that splits modes inside the high-quality region and can still fall back to a single answer at inference.
MoDA (Mode-conditioned Diversity Alignment) conditions one shared policy on abstract numbered roles. The system prompt is just "You are role i". Each role acts as an agent, emits one reply to the same user query, and the group is updated with GRPO. Hand-written personas are deliberately avoided so the diversity prior is not baked in. Ablations with crafted personas and dummy tokens such as "Start with APPLE" underperform numbered roles.
The reward has four parts:
Training uses 10k prompts, a 4:1 mix of Tulu3-SFT-Mixture and Infinite-Chat. The main base is Qwen3-8B; the reward model is Skywork-Reward-V2-Llama-3.1-8B-40M. Each prompt yields six role-conditioned samples. At inference, role tokens produce a diverse set; omitting them restores ordinary single-answer behavior.
On Qwen3-8B with thinking off, SBERT diversity on held-out Infinite-Chat rises 265% over the base model. Against the strongest training baseline DivPO, SBERT goes from 0.274 to 0.482 (+75.9%) and E-Vendi from 2.86 to 4.40 (+53.8%). Average general-capability pass@1 is 62.9 for the base, 66.2 for DivPO, and 73.2 for MoDA: +10.3 over the base and +7.0 over DivPO.
| Method | Infinite-Chat SBERT | Avg pass@1 |
| Qwen3-8B | base | 62.9 |
| DivPO | 0.274 | 66.2 |
| MoDA | 0.482 | 73.2 |
Per-task, MoDA scores GSM8K 86.7, GPQA 52.0, HellaSwag 73.4, TruthfulQA 57.9, all above the base; MMLU 81.0 and IFEval 76.9 sit near or slightly under it. On GLM-4-9B, average pass@1 moves from 44.9 to 60.7 (+15.8 points) and GSM8K from 21.8 to 84.9. In a qualitative check, 100 samples of "Name a shade of blue" yield 40 distinct shades, 122% more than the base and 150% more than the prompting method SSoT.
Ablations: role prompts without a diversity reward are not enough. Diversity-only training has the worst standard-mode accuracy and collapses under diverse decoding. A weighted sum of quality and diversity is brittle in λd: 1 under-diversifies, 10 and 100 hurt diverse-mode capability. The quality gate separates the two objectives and is less coefficient-sensitive.
This is drop-in online RL for existing post-training stacks. No architecture change. Turn roles off for a single reliable answer; turn them on for a set of hypotheses or drafts. For ideation, creative writing, and any setting where repeated samples only help if they differ, the training objective itself protects diversity. Capability numbers do not fall over to pay for that diversity, which is rarer than the method's MARL branding.
The quality reward model prefers verbose answers, and a uniform length penalty does not track prompt difficulty. On NoveltyBench, GLM-4-9B hacks the score with preambles such as "As a model"; stripping them lifts the native score from 2.739 to 3.302. Diversity cannot exceed what the initial policy can already express, because no external retrieval is added. Diverse mode can still emit more varied but less trustworthy text; high-stakes use should not treat that mode as the final answer. Medicine and law are not official evals. Quality on PreScience does not uniformly beat the baseline.