Mapping Similarity Spaces across Embedding Models with Synthetic Query Probing
Marcin Rozmus, Peter van der Putten
Science
cs.CL
2026-08-06
Similarity scores from different embedding models aren't comparable; Synthetic Query Probing generates labeled query-chunk pairs and learns isotonic mappings to align their score distributions for threshold portability.
RAG pipelines retrieve content by similarity score, but those scores live in different ranges depending on the embedding model. The same query-document pair might score 0.4 in Amazon Titan 256 and 0.83 in OpenAI's ada-002. That breaks two everyday engineering tasks: when you swap models, old recall thresholds no longer apply, and hard rules like "treat anything above 0.7 as relevant" silently stop working. The authors treat this as an under-studied measurement gap and set out to map it.
The key move is to align score distributions rather than embeddings. To do that you need many query-chunk pairs with known relatedness, which the authors generate with Synthetic Query Probing (SQP): from each chunk they synthesize questions at three relevance tiers, PARAPHRASE (answerable from the chunk), RELEVANT (topically related but needing more context), and IRRELEVANT. Using Claude Sonnet 4.6 they sample 100 chunks per corpus and generate 10 questions per chunk per tier, yielding 3,000 pairs per corpus with no human labeling.
With those pairs they fit a conversion function from model A's scores to model B's, comparing three forms: linear regression (OLS), isotonic regression (a monotone step function), and quantile mapping (a CDF-based percentile lookup).
The score distortion is systematic. On SciFact, Titan 256 spreads PARAPHRASE mean 0.54 and IRRELEVANT mean 0.06 apart clearly, while ada-002 crams them together at 0.86 and 0.69 with standard deviations two to six times smaller.
| Pair | Best converter | R² |
| Titan 512→1024 (same family, dimension change) | isotonic | 0.990 |
| Titan 1024→ada-002 (cross-model) | isotonic | 0.945 |
| ada-002→Titan 1024 (reverse) | isotonic | 0.889 |
Two clean takeaways. Dimension changes within a family are near-lossless and largely corpus-invariant (R²≥0.97). Cross-model mappings are noisier, non-linear, and corpus-dependent, and they are asymmetric: converting ada back to Titan has three to four times the mean absolute error of the forward direction.
Thresholds behave counter-intuitively. For 0.95 precision on SciFact, ada needs a threshold of only 0.718 and reaches 1.000 recall, while Titan 1024 needs 0.063 to hit 0.996. On the enterprise corpus the order flips, with Titan ahead on recall (0.76 vs 0.69). Thresholds track the corpus more than the model.
For anyone maintaining recall thresholds or planning an embedding swap, this gives two concrete rules: moving within a family to a higher dimension is safe and a linear map suffices; switching models requires non-linear recalibration, with isotonic regression as the default. SQP is also a cheap, label-free diagnostic you can rerun for migration checks and corpus-drift monitoring.
The authors list several: only two corpora, only four embedding configurations; synthetic queries may carry bias; the setup assumes normalized embeddings, static thresholds, and single-stage retrieval without reranking; and the conversions are fit on the full dataset with no held-out test set, so whether they work as reusable calibrators is open. A gap worth noting: the three relevance tiers are discrete, but real RAG relatedness is continuous, and the paper does not discuss how the binning behaves near boundaries.