Linkup open-sources SparseUp: a 149M sparse retriever hitting 56.4 nDCG@10 on BEIR
tomaarsen · x · 2026-09-18
Linkup released SparseUp on Hugging Face, a new sparse embedding model the author calls a rare breed in today's release landscape.
- Results: 149M params on a ModernBERT backbone, 56.4 nDCG@10 on BEIR-13 — claimed to be the strongest public vocabulary-based sparse encoder under 150M. Apache 2.0.
- Design: shares backbone and fine-tuning data with LightOn's DenseOn/LateOn, filling the sparse slot so all three architectures can be compared under controlled conditions; initialized from LateOn-unsupervised with ModernBERT's MLM head grafted on.
- Three key knobs: logitshift=15 (sparse-starting ReLU support), positiontopk=12 (per-token expansion budget), vocabfold to collapse case/space surface forms, shrinking vocab from 50k to 34k.
- Training: contrastive fine-tuning with 7 hard negatives sampled from 50 plus in-batch negatives; no cross-encoder distillation.
Related event: Linkup Open-Sources SparseUp: 149M-Parameter Sparse Retriever Tops BEIR-13(12 posts)→
More from Models
- 'You Can Feel the Gradient Slopes': Dev Contrasts Benchmaxxed RL Models vs General Ones — theteknosaur · 2026-11-02
- LightOnOCR-3 Launches: Tops FRBench, Adds Layout Structure and Chart Extraction — IgorCarron · 2026-11-02
- NVIDIA's Kumo Is a Family of Four Tabular Foundation Models, Not Just One — pandeyparul · 2026-11-02
- SemiAnalysis: safety classifiers are underrated — Meta's Muse tops cyber benchmark at 35.2% — ThomasScialom · 2026-11-02
- DeepSeek V4.1 Flash 1017 wins over a longtime closed-source-only user: fast, cheap, reliable — _bachrc · 2026-11-02
- Local quants vs Claude on 9 real private bugs: 3-bit costs thinking time, not score — FeydRowan · 2026-11-02