LambdaMix: one training run lets LLMs switch objective trade-offs at inference via LoRA mixtures

Constantinos Karouzos · hf · 2026-11-09

Researchers at the University of Sheffield introduce LambdaMix, a parameter-efficient post-training method that learns a preference-conditioned family of policies, addressing the limitation that standard post-training yields a single policy for one fixed objective weighting.

Method

Jointly trains one low-rank adapter per objective over a frozen base model, using group-relative online RL.
A single preference vector selects the adapter mixture for generation and weights separately normalized rewards for adapter updates.
One training run supports changing preference trade-offs at inference without further optimization.

Results

Tested on 4 models for helpfulness/harmlessness and math-with-instruction-following, plus comment rewriting that preserves wording while reducing personal-attribute clues on the two largest models.
On the first two tasks, LambdaMix achieves higher hypervolume than all conditioned baselines and policy-merging approaches.
It uses fewer training completions and fewer stored adapter parameters than a set of GRPO specialist policies.
Varying adapter mixtures and objective weights together yields wider score ranges and higher coverage than varying either alone.

Original post →

LambdaMix: one training run lets LLMs switch objective trade-offs at inference via LoRA mixtures

More from Research