Focusing Condition: Inference-Time Self-Contrastive Steering Elicits Better Conditional Text Embeddings in LLMs
Zifeng Cheng, Lingyun Qian, Zhiwei Jiang, Cong Wang, Yafeng Yin, Fei Shen, Ao Zhou, Qing Gu
ACL 2026 (Oral)
cs.CL
2026-09-26
SCS masks the condition in one extra MHSA pass and subtracts that view. On LLaMA3-8B-Inst it lifts PonTE's 7-dataset mean from 42.3 to 45.6 with near-zero extra latency.
The same sentence can be close or far from another depending on the aspect you care about. A man playing a guitar and a man playing a trumpet should match on gender and mismatch on instrument. A single universal embedding cannot do both.
Training-free extraction from LLMs currently stuffs the condition into a prompt. PonTE extends PromptEOL with the phrase in terms of [CONDITION] and reads the last token's hidden state. That vector still carries general text semantics. On TweetEmotion, the average cosine similarity between general and conditional embeddings is 0.763, so distance metrics get pulled by condition-irrelevant content. Small prompt edits also swing the scores.
Conditional text embeddings need a view of the text along one specified axis. A condition phrase in the prompt does not isolate that axis by itself.
Self-Contrastive Steering (SCS) is an inference-time plugin. No extra data, no fine-tuning; it attaches to existing prompt methods.
The two views differ only on the condition span, so subtraction highlights condition-related content. After masking, the prompt looks like unconditional PromptEOL. The position shift is required because RoPE would otherwise see a hole in the sequence.
Direct Elimination (DE) subtracts: delta v = vcond - vuncond. Projection-Based Elimination (PE) projects the conditional vector onto the unconditional direction and subtracts that component. The contrast changes the norm, so alpha rescales it.
The extra cost is one MHSA at a single layer. Typical activation steering needs labeled positive/negative pairs or a full second prompt. SCS builds a per-input steering vector from the same sequence and does not rerun the whole network.
Main table: LLaMA3-8B-Inst, seven datasets, clustering V-measure, STS Spearman/Pearson, triplet-alignment accuracy. PonTE has no public code; all PonTE-family numbers are reimplementations.
| Method | TweetEmo | I.STSB (Sp/Pe) | 7-task mean |
| PromptEOL | 31.7 | 0/0 | 22.7 |
| PonTE | 39.9 | 33.9/29.5 | 42.3 |
| PonTE+Att | 43.4 | 33.3/29.1 | 42.4 |
| SCS-DE | 45.5 | 39.9/37.7 | 45.6 |
| SCS-PE | 45.9 | 38.6/36.0 | 45.3 |
| E5-Mistral-7B-Inst | 41.3 | 34.8/34.6 | 7-task mean not reported |
Largest gain is I.STSB: PonTE 33.9/29.5 to DE 39.9/37.7 (+6.0/+8.2). Toxic only 57.7 to 58.0 (+0.3), the weakest of the seven. Best or tied-best on six tasks; on AGNews, PE at 85.8 ties PonTE+Att. DE's mean slightly beats PE, so projection is optional.
On Amazon-C, SCS-DE scores 34.2 against E5-Mistral-7B-Inst at 37.4; the supervised model still leads by more than 3 points. Clustering is split. On a V100 with matched batch size, per-sample time is 70ms vs 69ms on TweetEmo and 79ms vs 80ms on Amazon-R. The extra MHSA sits inside noise.
On base LLaMA3-8B, the three-task mean goes from 17.4 to 28.0 (+10.6); the instruction-tuned variant from 34.5 to 40.0 (+5.5). Qwen2.5-7B base +3.2, instruct +4.0. Weaker instruction following leaves a dirtier prompt embedding, so contrast helps more. Gains hold across six prompt templates and six condition phrasings, and prompt variance drops. Best intervention layer is dataset-specific: layer 3 on TweetEmo, 7 on Amazon-R, 4 on Amazon-C. Alpha is grid-searched in {5, 10, 15} per dataset.
If you already pull embeddings from an open LLM with PromptEOL or PonTE, and you can edit the attention mask, this drops in. No training set, and the model stays a generator. You pay one extra attention layer and two hyperparameters.
This is an incremental boost, not a new task. Gains concentrate on short text, fine-grained STS, and emotion clustering. Triplet alignment barely moves. Anyone shipping this has to search layer and alpha per dataset, and has to own the weights.
The paper states two: internal intervention, so black-box APIs are out; no tests on 70B-class models.
The experimental protocol is a bigger caveat. Layer l is searched in [2, 20] and alpha in {5, 10, 15} per dataset; the three clustering sets disagree on both. Hyperparameters do not transfer across tasks. Intervention site is not shared either: FFN wins on I.STSB (40.8/39.1), hidden states win on TweetEmo (46.2). Toxic's +0.3 suggests that when the prompt already encodes the condition well, contrast has little left to subtract.
PonTE numbers are reproductions. Steering also moves the best readout layer: PonTE peaks at layers 30 and 29, SCS at 32 and 28. That layer is another search.