Single-Layer Self-Contrastive Steering Improves LLM Conditional Embeddings at Inference Time

_reachsumit · x · 2026-09-29

A new arXiv paper introduces Self-Contrastive Steering (SCS), a plug-and-play inference-time method for better conditional text embeddings from LLMs.

Problem: prompt-based conditioning leaves conditional embeddings entangled with general ones, capping quality.

Method and results:

SCS masks the condition via modified attention masks and positional encodings to obtain unconditional embeddings, then steers the multi-head self-attention computation with them, making conditional embeddings more focused.
No extra data or fine-tuning needed; overhead is a single additional multi-head self-attention pass at inference.
On clustering, Semantic Textual Similarity, and triplet alignment benchmarks, SCS consistently improves existing prompt-based methods. Code released.

More from Research