LightOn open-sources SOTA sparse retrievers SlideOn and mSlideOn at 174M and 339M parameters
IgorCarron · x · 2026-10-19
LightOn's R&D team released SlideOn and mSlideOn, fully open-source SOTA sparse retrievers at 174M and 339M parameters, betting on sparse retrieval after the dense and late-interaction waves.
- Accompanying blog post is unusually detailed: how to properly train dense or late-interaction models, how to fix bad SAE models built on top of late-interaction, and evidence that SAEs share the multilingual generalization of late-interaction models and excel at code retrieval.
- Not as sparse as it could be yet, but the team provides out-of-the-box ways to sparsify further.
Model weights and blog post are publicly available.
Related event: LightOn Open-Sources SOTA Sparse Retrievers SlideOn and mSlideOn(8 posts)→
More from Research
- Forward-only pretraining matches backprop: 3.704 val in 2.5 min on 8×H100, no autograd — tensorqt · 2026-10-20
- NeurIPS 2026 paper 'Intelligence per Watt' measures on-device AI efficiency across 1M queries — HazyResearch · 2026-10-20
- Cohere Labs talk: Goodfire's Matthew Kowal on interpretability across the model design pipeline — Cohere_Labs · 2026-10-20
- NeurIPS paper: AI models can provably hide arbitrary capabilities in encrypted weight backdoors — MoonL88537 · 2026-10-20
- 26,811-pupil study: AI lifts homework scores in China but hurts exam performance — Dr_Alex_Crimi · 2026-10-20
- Dev post-trains Gemma 4 E2B into a decision model, 20x faster and beats local rivals — andrew_n_carr · 2026-10-20