CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity
Ananya Sahu, Mohit Bansal, Elias Stengel-Eskin
cs.CL, cs.AI
2026-08-08
CreativeInstruct fine-tunes one aligned LLM to self-inject creativity tokens, restoring diversity (70.3% human preference) and, as a GRPO start point, lifting math reasoning ~5 points.
Alignment makes models stronger and duller. After SFT and RLHF, outputs cluster into a few stable modes, a property the literature calls homogeneity. Repetitive story beats are the visible symptom; the subtler cost shows up in reinforcement learning. Algorithms like GRPO sample several rollouts of the same problem and compare them, and when those rollouts look alike the policy has nothing to choose between and keeps drilling one solution.
Existing fixes such as BACo route per token between a base model and an aligned model at inference time, borrowing the base model's spread where the aligned model is too tight. That costs a second model on every request, adding latency and memory, and it needs the base model, which for families like Qwen3 32B is never released. CreativeInstruct distills that routing decision into one model: borrow BACo to build training data, then run a single model at inference that decides for itself when to loosen up.
The mechanism turns BACo's routing into a learned skill, in two stages.
Stage one builds data. From the Tülu V3 SFT set, 4,000 English writing prompts are filtered, with three outputs each (12,000 total). During generation BACo routes tokens by entropy and sentence boundaries: high-entropy tokens and sentence-edge tokens are more likely to come from the base model. Spans produced by the base model are wrapped in [StartCreativity] / [EndCreativity] tokens, tagging "this is where the creativity came from."
Stage two fine-tunes. The aligned model is LoRA-tuned (r=32, on attention and MLP) on this tagged data. What it learns is timing (where to insert [StartCreativity]), not content. At inference the model self-injects the marker and the following span drifts toward base-model variety. One model, no base model needed, no dual-model routing overhead at test time.
The paper also introduces a new metric, LLM-GED. An LLM extracts an abstract narrative graph per story: nodes are entities and events, edges are semantic relations and temporal order, with roles abstracted to placeholders like Character1. A normalized graph edit distance is averaged across story pairs. The motivation is that lexical metrics (cosine) and semantic metrics (NLI diversity) miss plot-level sameness; two stories told in different words but with the same arc look distinct to those metrics but not to LLM-GED.
Diversity on LLaMA-3.1 8B (Table 1):
| Method | Cos-D (M) | Vendi | NLI Div | LLM-GED |
| Instruct | 0.309 | 2.913 | 0.040 | 0.366 |
| BACo | 0.255 | 3.499 | 0.065 | 0.374 |
| CrPO | 0.350 | 3.852 | 0.028 | 0.514 |
| Distill | 0.444 | 3.975 | 0.052 | 0.523 |
| CreativeInstruct | 0.458 | 4.749 | 0.085 | 0.545 |
Cos-D is a cosine diversity score and Vendi is distributional diversity; higher is better on both. CreativeInstruct leads on nearly every column. The authors report roughly 48% relative gain in semantic diversity and 63% in structural diversity over the instruct baseline. Proper-noun uniqueness tells the same story: within a single prompt, CreativeInstruct uses unique names 37.1% of the time versus 18.1% for the instruct model, which keeps grabbing the same names.
Human evaluation (n=50): annotators preferred CreativeInstruct for creativity in 70.3% of cases (binomial test significant) and for diversity in 57.4% (not significant). Quality was dropped from the human study because annotator agreement was too low (κ=-0.167); the paper relies on automatic metrics there.
The RL result is the load-bearing one. On Qwen3 8B, 1,000 steps of GRPO, comparing RL on a CreativeInstruct checkpoint against RL on the original aligned checkpoint:
| Setup | MATH | AMC |
| Instruct baseline | 0.374 | 0.432 |
| Instruct + RL | 0.409 | 0.438 |
| CreativeInstruct baseline | 0.424 | 0.428 |
| CreativeInstruct + RL | 0.459 | 0.478 |
Same GRPO, more diverse start: +5 points on MATH (0.459 vs 0.409) and +4 on out-of-distribution AMC (0.478 vs 0.438). A more diverse policy gives RL more to explore, and it generalizes better to problems it never saw.
For anyone doing RL post-training, the practical takeaway is that the diversity of the checkpoint you start GRPO from affects where you end up. People tune reward functions and KL coefficients; the spread of the initial policy's rollouts is an overlooked variable. If RL is stuck on a plateau, a more exploratory starting policy may help more than another hyperparameter sweep.
For generation work, it compresses BACo's two-model inference-time routing into one model with no test-time overhead, and it works without the base model (train on data generated by another model). The cost is a LoRA pass plus needing a base model at data-construction time, but that dependency moved from inference to offline.
This is a solid incremental contribution: a simple method (special tokens plus LoRA), a metric (LLM-GED) that fills a real gap, and an experiment that links diversity to reasoning.
The authors flag several. Qwen3 32B has no public base model, so they trained on data from Qwen2.5 32B, a transfer shown on a single model with generalization unproven. Quality human agreement was too low (κ=-0.167), so the central promise of "without sacrificing quality" rests entirely on automatic reward models rather than human judgment.
Two more concerns stand out on close reading. The RL gain is shown on one model (Qwen3 8B), one domain (math), and 1,000 steps; five points is modest, and whether it holds for code or larger models is open. And diversity scores had not plateaued at 12,000 samples, which the authors read optimistically as room to grow but which also means the reported numbers are not a stable ceiling.