Steerable Multi-Objective Group-Relative Preference Optimization with Jointly Trained Low-Rank Adapters
Constantinos Karouzos, Nikolaos Aletras
cs.CL, cs.AI, cs.LG
2026-11-06
LambdaMix uses one preference vector for both LoRA adapter mixing and reward weighting; a single GRPO run beats conditioned and merging baselines and supports steerable trade-offs at inference.
Alignment often juggles competing objectives: helpful answers push against safety refusals, and math problems come with format constraints that fight free-form reasoning. Standard RLHF or GRPO bakes in one fixed reward weighting and produces a single policy; when preferences shift after deployment, the usual answers are either expensive or weak. Merging separately trained models (rewarded soups, Preference Vector) or routing between adapters (HoE) needs multiple training runs. Prompt-level conditioning like CLP trains once but barely responds: on Qwen3.5-9B, sweeping the preference weight moves its helpfulness score by only 0.300. The goal here is one training run that yields a whole family of policies, with a single inference-time knob.
LambdaMix freezes the base model and attaches one LoRA adapter (rank 8) per objective. A preference vector λ on the simplex sets the layer weights directly: W(λ) = W0 + (α/r)·Σ λt·Bt·At. B starts at zero, so the initial policy equals the base model for every preference. One λ does two jobs:
Each mixture policy therefore learns toward the weights that match it. Ablations on Qwen3.5-9B show the two jobs must move together: fixing either the mixture or the weights drops hypervolume from 0.8735 to 0.8144 and 0.7960, and shrinks score spans from 1.931/1.982 to the 0.1 range, which kills steerability. At inference, pick a λ, mix the adapters, generate. No reward models, no updates, and a fixed preference can even be folded into the base weights ahead of time.
Three tasks, two objectives each: helpfulness versus harmlessness on PKU-SafeRLHF (scored by Beaver reward and cost models), correctness versus instruction compliance on GSM8K with format constraints, and wording retention versus attribute-clue removal on SynthPAI rewrites. Four base models on the first two tasks; the two largest on the third.
| Setting | Metric | LambdaMix | Best baseline |
| PKU / Qwen3.5-9B | hypervolume | 0.8735 | 0.7760 (prompt conditioning) |
| PKU / Qwen3.5-9B, λ=(0.5,0.5) | helpfulness / harmlessness | 5.586 / 8.487 | 5.380 / 8.164 (prompt conditioning) |
| PKU / Qwen3.5-9B | score spans | 1.931 / 1.982 | 0.300 / 0.379 (prompt conditioning) |
| GSM8K / Qwen3.5-9B | hypervolume | 0.8099 | 0.6947 (GRPO specialists) |
| GSM8K / Qwen3.6-27B | hypervolume | 0.8895 | 0.8299 (GDPO specialists) |
| SynthPAI / Qwen3.6-27B | hypervolume | 0.4552 (normalization off) | 0.4407 (GRPO specialists) |
On the two larger Qwen models for PKU, every evaluated baseline setting is dominated by at least one LambdaMix setting. The GSM8K sweep on Qwen3.6-27B is fully ordered: raising the accuracy weight moves XML accuracy from 0.470 to 0.953 while compliance falls from 0.978 to 0.745, with zero reversals. On cost, one run with 32,000 completions and 13.2M adapter parameters supports any preference; a GRPO or GDPO specialist set needs five runs, 160,000 completions, and 33.1M parameters for five fixed preferences. At λ=(0.5,0.5), LambdaMix beats all ten specialists on both scores at once. The exceptions are real: HoE wins on Olmo 3 7B for both PKU (0.7844 vs 0.7625) and GSM8K (0.5581 vs 0.5291). Per-objective normalization helps on all four PKU models, splits on GSM8K (two up, two down), and hurts on both SynthPAI models; choosing it is left to trial and error.
For teams serving user groups with different safety appetites, this replaces a rack of specialist checkpoints with one checkpoint and a dial: 80% less training volume (32k vs 160k completions, one run vs five), 60% fewer adapter parameters (13.2M vs 33.1M), and a score span about six times wider than the strongest conditioned baseline (helpfulness 1.931 vs 0.300 on Qwen3.5-9B). The components are not new: preference-conditioned LoRA mixing comes from PaLoRA, group-relative training from GRPO, per-objective standardization from GDPO. The new piece is letting one λ drive both the generation side and the update side inside online RL, with controlled ablations showing those two must move together. Code is public and all base models are Apache 2.0, so reproduction cost is low. This is a solid compositional improvement, not a new optimization principle.
The authors' own list: only two objectives are tested, extending further and decoupling the two λ roles are future work; SynthPAI scores do not establish protection against attribute inference, and learned evaluators can inherit biases; steerability cuts both ways, since users can dial harmlessness down, so preference ranges should be restricted before deployment. A few more concerns from a close read: helpfulness and harmlessness are measured by the same Beaver learned reward models used in training, so reward hacking is unmeasured; all baselines were re-implemented by the authors inside the group-relative setup, so tuning parity is not guaranteed; training lasts only 500 optimizer updates, and behavior at larger budgets is unverified; normalization helps or hurts by task and model, with a variance-based hypothesis but no selection rule.