Linkup open-sources SparseUp: a 149M sparse retriever hitting 56.4 nDCG@10 on BEIR
tomaarsen · x · 2026-09-18
Linkup released SparseUp on Hugging Face, a new sparse embedding model the author calls a rare breed in today's release landscape.
- Results: 149M params on a ModernBERT backbone, 56.4 nDCG@10 on BEIR-13 — claimed to be the strongest public vocabulary-based sparse encoder under 150M. Apache 2.0.
- Design: shares backbone and fine-tuning data with LightOn's DenseOn/LateOn, filling the sparse slot so all three architectures can be compared under controlled conditions; initialized from LateOn-unsupervised with ModernBERT's MLM head grafted on.
- Three key knobs: logitshift=15 (sparse-starting ReLU support), positiontopk=12 (per-token expansion budget), vocabfold to collapse case/space surface forms, shrinking vocab from 50k to 34k.
- Training: contrastive fine-tuning with 7 hard negatives sampled from 50 plus in-batch negatives; no cross-encoder distillation.
Related event: Linkup Open-Sources SparseUp, a Sub-150M Sparse Retriever Topping BEIR-13(11 posts)→
More from Models
- Blogger claims DeepSeek v4.1 flash is why calls to pace the frontier emerged — idanbeck · 2026-09-18
- Steve Yegge confirms unannounced partial reset of Claude Max $200 weekly limits — Steve_Yegge · 2026-09-18
- DeepSeek-V4.1-Flash lands on Nebius: 552B MoE activating 8B with 1M context — Arindam_1729 · 2026-09-18
- Anthropic opens Life Sciences Verification Program, grants verified teams access to Mythos for bio research — AllThingsApx · 2026-09-18
- Is the Hype Around Type Safe AI's Jev and RLCD Justified? A Skeptic's Take — Haghiri75 · 2026-09-18
- Yale Psychologist Puts Claude Opus On Peer Review Duty, Catches Errors Human Reviewers Miss — lu_sichu · 2026-09-18