SPARSEUP: a recipe that puts modern backbones to work as sparse retrievers, 56.4 nDCG@10 on BEIR-13

SPARSEUP: Towards Better Recipes for Training Modern Learned Sparse Retrievers

Thibault Formal, Pierre Erbacher, Cosimo Rulli

cs.IR

2026-11-07

Logit shifting, per-position fan-out caps, and vocabulary folding make Splade-style sparse retrievers trainable on modern backbones; SPARSEUP reaches 56.4 nDCG@10 on BEIR-13, best under 150M parameters.

What problem this solves

Learned sparse retrieval represents each document as a sparse vector over the vocabulary: a term carries weight only if the model activates it, retrieval runs on an inverted index, and rare-term matching stays exact. The Splade series set the standard here for years, then stalled. Dense single-vector and late interaction models kept gaining from newer backbones and data, while sparse encoders did not transfer: ModernBERT and Ettin underperform under the standard Splade recipe, and RoBERTa-Large simply fails to train.

Two recent papers diagnosed this differently. Geng and Yang blame the vocabulary: byte-level BPE tokenizers keep case and word-boundary variants as separate entries, spreading one word over up to four surface forms. Jang et al. blame scale: large-norm LM heads inflate activations at initialization. This Linkup technical report takes the position that both diagnoses hold, and that one simpler recipe addresses them together.

Method

Three ingredients, all acting on the model itself, with no inference-time compensation. Training follows the DenseOn/LateOn contrastive setup (InfoNCE, temperature 6, 7 hard negatives per query).

Applied to the LateOn backbone (with ModernBERT's MLM head grafted on) and the Embeddings-Fine-Tuning dataset, this yields SPARSEUP, under 150M parameters.

Results

Average nDCG@10 on BEIR-13:

ModelTypeBEIR-13
Splade-v3sparse51.7
OS-NSE doc-v3sparse54.6
SPARSEUPsparse, <150M56.4
DenseOndense, same backbone/data57.9
SlideOnlate interaction58.7
Laconicsparse, 1B58.7
LateOnlate interaction58.9

56.4 is, per the authors, the highest they know of among public vocabulary-based sparse encoders below 150M parameters, +1.8 over the strongest comparable model. It lands 1.5 points behind DenseOn trained on the same backbone and data, and on the decontaminated BEIR (leaked queries removed) the two are essentially tied at 57.5 vs 57.7. It beats DenseOn and LateOn on ArguAna and Touché. Retrieval runs on a Seismic index in under 1 ms per query.

Ablations: removing vocabulary folding drops the average to 55.0; starting from bare ModernBERT instead of the LateOn checkpoint drops it to 54.3. Cross-encoder distillation adds only 0.1 on BEIR-13 but pays off out of domain: +2.5 on CURE-en, +2.3 on Legal/Finance, +3.3 on ChemTEB, and SPARSEUP without distillation already beats DenseOn by 4.9 nDCG@10 on BRIGHT-Pro. One practical finding: if you will deploy at heavy pruning anyway, training with a top-64 constraint beats the unconstrained model by 4 points at that budget, though such models degrade at larger k. LoTTE is the main exception, where sparse trails all dense and late interaction models.

Why it matters

For anyone building retrieval or RAG systems, the message is that sparse retrieval is a live option again. The years-long blocker, modern backbones failing to train as sparse encoders, is fixed by three light changes: a constant subtraction, a top-k before pooling, and a parameter-free merge. On effectiveness this is a catch-up, not an overtaking: same-setting dense and late interaction models still average higher. What sparse buys is sub-millisecond queries on mature inverted-index infrastructure, exact rare-term matching, interpretability, and an edge on several out-of-domain suites like BRIGHT-Pro, legal/finance, and chemistry. Near-dense quality at roughly an order of magnitude cheaper serving is often the right trade in production. Weights are public as linkup-sparseup-embed-v1.

Limitations

The authors' own list: wacky weights persist, with sub-tokens activating irrelevant fragments (cle firing cre, ce, Cole, cane), so representations look less clean than Splade-v3's even while scoring higher; LoTTE trails all dense and late interaction baselines; and Laconic, roughly an order of magnitude larger, sits 2.3 points ahead. The Embeddings-Fine-Tuning training set contains training splits of some evaluation collections; the decontaminated comparison partially covers this, but the headline average deserves a discount.

Open questions after reading: the 99.9th-percentile rule is validated on three backbones and may not transfer blindly; f=12 is an empirical sweep; the headline number depends on the LateOn pretrained checkpoint, and without it the model drops to 54.3; all results are single runs with no variance reported; and best-below-150M is a self-scoped claim, with larger and distilled models outside the comparison.

Terms

Source

What people are saying

Related papers

All paper explainers