LightOn open-sources SlideOn: SAE-based sparse retrieval beats its dense teacher at 174M params

tomaarsen · x · 2026-10-22

LightOn released the SlideOn family (plus multilingual mSlideOn), converting dense retrieval models into sparse ones via sparse autoencoders (SAE):

SlideOn starts from DenseOn, with SAE pretraining followed by supervised retrieval fine-tuning; it emits 32,768-dim nonnegative sparse vectors (top-32 latent activations + ReLU, max-pooled over tokens) compatible with inverted-index search;
It reaches 56.68 nDCG@10 on BEIR-15, beating DenseOn's 56.20;
Joint query/document Matryoshka training preserves quality under pruning, so search/storage costs can be tuned at inference without retraining;
mSlideOn starts from mLateOn for multilingual, long-context and code retrieval;
Models, SAE initializations, data recipe and ablations are all Apache-2.0 open source, with a full blog post.

More from Models