When Direction Matters: Diagnosing Relational Instruction Following in Dense Retrieval
Zhanyu Wu, Richong Zhang
cs.IR
2026-11-11
SwitchIR freezes query and documents, flipping relation direction. Seven retrievers score 0.986 FlipBoth on constraints but 0.015 ContrastAll on flips, 0.084 on realistic text.
Instruction-tuned retrievers are supposed to change their ranking when a user adds a sentence of relevance criteria. On FollowIR and InstructIR they often do, provided the right document carries a different audience, attribute, or negation. In those items the two passages are easy to separate by their words.
A complaint that caused a redesign, versus a redesign that caused a complaint, is a different case. The entities and the content words barely move, and the passage that should rank first swaps. Fold that role binding into an attribute-filtering average and the miss disappears. Dense retrievers, which place the query and the document in one vector space and rank by similarity, already sit inside RAG. A reversed relation can be ranked wrong while the leaderboard stays quiet.
SwitchIR is a pairwise test. Each group freezes one topic query and two mutually hard-negative documents, passages close in topic and easy to confuse, and changes only the instruction. A tie counts as wrong. Both candidates are always present, so a miss is a preference error rather than a failure to surface the passage from a corpus.
PairAcc averages the two anchor decisions. Chance is 0.5. FlipBoth requires both opposite anchors to be right. Chance is 0.25. ContrastAll requires every paraphrase of both directions. Chance is 0.0625, and that is the headline number for Relational Flip. Constraint Shift has one instruction per side, so FlipBoth is its joint score.
The controlled set locks wording with templates: 700 groups and 2,700 instructions. The realistic set is an LLM rewrite that the authors checked group by group: 140 groups and 520 instructions. The two sets are reported separately. Every condition from a group stays in the same split. Embedding models use cosine similarity. BGE-M3-ColBERT uses late interaction, matching tokens and then pooling. BM25 is lexical. Qwen3 rerankers score the query and the document jointly. All of this is zero-shot, with each model's own instruction template.
On Relational Flip the margin of four embedding models, Qwen3-Embedding 0.6B, 4B, 8B, and GritLM, splits exactly into a direction-invariant pair preference b and a direction-sensitive interaction r. The two margins are b+r and r-b. Both directions are correct only when r exceeds |b|. The interaction is the product of the query shift, the document shift, and the alignment between them.
Across seven neural retrievers, excluding BM25 and the rerankers, the controlled means are FlipBoth 0.986 on Constraint Shift, ContrastAll 0.923 on Explicit Filtering, and ContrastAll 0.015 on Relational Flip. The last number is below the 0.0625 chance level for a joint decision. PairAcc on the flip is still 0.49 to 0.59, and FlipBoth, both anchors correct, is only about 0.14 to 0.27.
| Setting | Constraint Shift FlipBoth | Explicit Filtering ContrastAll | Relational Flip ContrastAll |
| Seven-model mean, controlled | 0.986 | 0.923 | 0.015 |
| Seven-model mean, realistic | 0.943 | 0.679 | 0.084 |
| Promptriever, controlled | 1.000 | 1.000 | 0.000 |
| Qwen3-Rerank-4B, controlled | 1.000 | 1.000 | 0.292 |
| Qwen3-Rerank-4B, realistic | 1.000 | 1.000 | 0.470 |
| BM25, controlled | 1.000 | 0.000 | 0.000 |
| BM25, realistic | 0.000 | 0.000 | 0.000 |
On realistic text the same seven models average 0.943, 0.679, and 0.084. Promptriever is perfect on Explicit Filtering and scores 0 on Relational Flip. Qwen3-Rerank-4B reaches 0.470 on realistic flips, 0.53 short of its perfect Explicit Filtering score. BGE-M3-ColBERT's controlled Relational Flip ContrastAll is 0.003. Qwen3-Embedding 0.6B, 4B, and 8B score 0.003, 0.000, and 0.020, so scaling the bi-encoder leaves the gap in place. The reranker moves from 0.105 at 0.6B to 0.292 at 4B. BM25 is perfect on controlled Constraint Shift and zero once the word inventory matches; after the realistic rewrite its Constraint Shift FlipBoth is zero as well. The best realistic Relational Flip score among non-rerankers is GritLM at 0.180.
FollowIR p-MRR measures how far full-corpus ranks move when the instruction changes. Promptriever leads that table at 12.8 and scores 0 on Relational Flip. E5-Mistral has FollowIR p-MRR 0.0 and the highest embedding ContrastAll in that comparison, 0.045. The two diagnostics order the models in opposite ways.
On every cell of the four embedding models, mean |b| is larger than mean r. An aligned signal strong enough to win appears in 14.2% to 25.5% of controlled observations and 13.5% to 32.5% of realistic ones. Aligned but too weak is more common: 28.8% to 36.9% controlled, 45.5% to 58.5% realistic. Each model contributes 1,200 controlled observations and 200 realistic ones.
The fine-tuning numbers below are ContrastAll in percent. LoRA on Qwen3-Embedding 0.6B and 4B uses controlled data only and a pairwise ranking loss. Trained on one relation family, the 4B model averages 99.0% on that family and 28.1% on the others; realistic scores fall to 65.2% and 26.4%. Trained on 350 groups from five families and tested on the held-out family, 4B reaches 92.5% controlled (sample SD 6.6) and 70.1% realistic (sample SD 3.9). The 0.6B model reaches 35.7% and 13.5%. Realistic held-out scores for 4B run from 17.6% on containment or hierarchy to 97.5% on directed action.
Many instructions that reach a RAG context are directional: which event caused which, which came first, who passed what to whom. A high score on attribute constraints says nothing about those. FollowIR and Relational Flip disagree on which retriever is better, so a model chosen from an existing instruction benchmark can still fail here.
The probe needs no index, only two passages. Zero-shot embedding models, at 0.015 ContrastAll, are a poor fit for opposite directions. A 4B reranker reaches 0.470 on realistic text. After multi-family tuning the 4B encoder transfers to an unseen relation; the 0.6B encoder's realistic held-out score is 13.5%. The failure has a sharp form. The direction signal has to beat the model's standing preference for one of the two documents.
SwitchIR covers reversed semantic roles. Nested constraints, multi-hop relations, and several conditions acting together are outside that scope.
The candidates are always a given pair. In a full corpus the reversed document may never reach the top ranks, and a ContrastAll near zero is not evidence about nDCG. If both passages are already in the generator context, the error is read straight through. That setting is untested. The margin split covers four bi-encoders. Fine-tuning covers two Qwen3 sizes with LoRA, and realistic text is only an external test. The 92.5% held-out figure is on controlled templates; realistic text drops it to 70.1%, and containment or hierarchy sits at 17.6%. A joint score below the chance baseline comes from a stable document preference. Whether the same split still governs a long candidate list is open.