Geometry of Values: Task Vector Composition for Ethical Preference Alignment in Language Models
Utkarsh Agarwal, Monojit Choudhury
ICML 2026
cs.CL, cs.AI
2026-09-18
Llama-3.2 1B/3B hit >98% on 12k ethical dilemmas; orthogonalizing task vectors against instruction-following then reversing them flips the stance, retaining ~98% of LoRA on 3B.
Content moderation, policy drafting, and personal advice all push a model into forced choices where both options are virtuous. Prior alignment work mostly probes stereotypes and toxicity. Direct clashes among Honesty, Justice, and Autonomy are less tested. Writing a policy into the prompt looks like it steers the model until the language or option order changes.
A sharper question sits underneath the accuracy numbers. After fine-tuning, did the model learn an abstract ranking such as A over B, or did it memorize wording in the templates? If the ranking is real, it should live in weight space and reverse under arithmetic. If it is template correlation, a new set of stories will collapse it. This MBZUAI workshop paper asks two things of compact Llama-3.2 1B and 3B models: can they learn an implicit value preference from pairwise dilemmas, and can that preference be isolated as a task vector and flipped without retraining.
Three principles only: Honesty/trustworthiness (A), Justice/fairness (B), Autonomy (C). That yields three pairwise tasks: Honesty vs Justice (AB), Justice vs Autonomy (BC), Autonomy vs Honesty (CA). Each item is a scenario plus two options. Both options are defensible; there is no ground truth until a stance is fixed. Under S = (A ≻ B), the A-aligned option is the target.
GPT-4o generated the items from 20 hand-written social settings, 4,000 per pair, 12,000 total, split 3200/400/400. Gemini-3-Flash-Preview edited 4,055 of 12,000 samples (label fixes, clearer conflicts, formatting). Google Translate produced Hindi, Arabic, Spanish, and Chinese parallels. The authors treat these as linguistic variants, not culturally localized dilemmas. Six people wrote candidate items; 60 survived as a human gold set to test template overfitting.
Two evaluation tracks. GPT-5-mini is queried zero-shot for default leanings, then steered with “always value A over B.” Llama-3.2-1B/3B get LoRA SFT and DPO on Wq, Wk, Wv only (r=4, α=8 on 1B; r=16, α=16 on 3B; lr 2e-4; 5 epochs). DPO uses β=0.1 over three seeds. At test time the stance is omitted; the model must return A or B. Option order is randomized.
The task-vector trick splits “can follow the output format” from “which value wins.” Naive negation, ΔS = θS − θ0 then add −ΔS back, produced garbage: reversing the whole delta also reversed instruction following. The working recipe:
γ1 and γ2 are grid-searched on the dev set (coarse step 0.3, then 0.1). Useful values sit around γ1 ∈ [0.3, 0.8] and γ2 ∈ [0.3, 0.7].
They also test transitivity: given A≻B and B≻C, does adding the preference vectors yield A≻C. That is the actual geometry claim.
With no policy, GPT-5-mini leans the same way in all five languages: Justice over Autonomy, Honesty over Autonomy. BC picks Justice about 70% of the time (English 69.5%, Hindi 73.5%). CA picks Autonomy only about 30%, so Honesty beats Autonomy by the same margin. Honesty vs Justice is a weak Honesty lean (English 56.8%, Hindi 64.5%).
Prompted stances reverse most of those biases, except Hindi and the Honesty-Justice pair. English AB hits 85.8% on the default order and 72.8% on the reverse; Hindi falls to 61.8% and 57.0%. Hindi AB even slides backward: the unsteered Honesty rate is 64.5%, and telling the model to keep preferring Honesty drops accuracy to 61.8%. Llama-3.2 1B and 3B are near chance and lock onto the first option (sometimes the second, mainly in Arabic).
Five-epoch LoRA SFT wipes the position bias and reaches ≥98% on the synthetic test set across languages and pairs. The same checkpoints stay above 90% on the 60-item gold set, with many BC cells at 100%. DPO tracks SFT with a small drop.
| Setup | Metric | Result |
| Llama-3.2 SFT | Synthetic test accuracy | ≥98% (5 languages, 3 pairs) |
| Llama-3.2 SFT | 60-item human gold | >90%, BC often 100% |
| GPT-5-mini, no policy | BC picks Justice | 70% across languages |
| GPT-5-mini, prompted | Hindi AB default / reverse | 61.8% / 57.0% |
| 3B task-vector reversal | Share of full LoRA accuracy | ≥98% on most cells; Hindi AB 85% |
On 1B, reversal keeps ≥93% of fine-tune accuracy on BC and about ≥80% on AB/CA. On 3B it keeps ≥98% on two pairs and 85% on Hindi AB. Which pair supplies the instruction vector barely matters. Transitivity fails: after a full grid search the composed models are near random. Preference vectors for different stances sit at angles above 80°, nearly orthogonal in weight space, so they do not chain into a third edge.
For anyone who wants switchable values on a small on-device model, this is cheaper than storing several checkpoints. Train one LoRA per binary stance, then add or subtract at inference. On 3B, most reversals keep 98% of the fully fine-tuned score, which is evidence that a preference direction can be peeled off instruction following.
For alignment research the claim is narrower and harder. Prompt steering is brittle even on GPT-5-mini, especially in a lower-resource language. LoRA on 1B/3B is enough to learn a forced-choice stance that survives a human gold set. The “geometry of values” supports local reversal only. A≻B plus B≻C does not produce A≻C. The title’s geometry stops at the flip; composition is still missing.
This is incremental work on three principles, binary choice, and 1B/3B. The reusable piece is the orthogonalization step, not a general moral compass.
The authors list five. The benchmark scores stance adherence, not moral reasoning or explanations. Most data is model-generated and machine-translated; the gold set has 60 items, so transfer evidence is preliminary. All PEFT and vector work is on Llama-3.2 1B/3B. The method is not training-free: it needs paired fine-tuned checkpoints and a dev search for γ.
A few gaps sit outside that list. GPT-4o wrote the corpus and Gemini rewrote a third of it; human spot-checks still found misaligned options and labels. Passing gold does not prove the model ignored generator style. Translations are not cultural rewrites, so cross-lingual numbers test language transfer. The transitivity failure is blamed on orthogonal vectors; the three principles may also fail to form a transitive chain in the data, and the paper does not separate those stories. Inference reads a single A/B token with no chain of thought, so what is learned looks like a classification boundary.