Search researcher on MRL debate: losses are tolerable but benchmarks are too small with stale QRELs
srchvrs · x · 2026-10-26
Search researcher srchvrs responds to ZeroEntropy's blog claiming Matryoshka Representation Learning (MRL) is lossy: he thinks the loss is real but small enough to tolerate.
He adds two structural flaws in current embedding benchmarks:
- Too small: the larger the document count, the more visible embedding regressions become — NanoBEIR is particularly bad here;
- Stale QRELs: old relevance judgments were built around BM25 (+ re-ranking), so many relevant documents surfaced by newer models aren't included, masking regressions.
This reinforces ZeroEntropy's point that public benchmarks are insensitive to nuanced regressions between frontier models.
Related event: Retrieval researcher challenges 'lossless' claims for MRL and ANN search(2 posts)→
More from Research
- lm-watermarking: official open-source implementation of LLM text watermarks — adnan_hashmi · 2026-10-26
- Kenneth Stanley on why measuring open-endedness is so hard for AGI — mhmazur · 2026-10-26
- Leicester researchers team up with Carelight on AI wearables to spot stroke changes earlier — tbirdcymru · 2026-10-26
- Hugging Face Journal Club derives the math behind score centering for off-policy RL — Hugging Face · 2026-10-26
- EMNLP paper: tweaking multiple-choice scoring schemes reshuffles LLM leaderboards — boydgraber · 2026-10-26
- Trustworthy AI pioneer Nicolas Papernot wins 2026 Arthur B. McDonald Fellowship — VectorInst · 2026-10-26