SlideOn model card details SAE pretraining and Matryoshka sparse training recipe
tomaarsen · x · 2026-10-20
The HF model card details the recipe: SlideOn starts from DenseOn, maps token representations to top-32 latent activations with ReLU, and max-pools into 32,768-dim sparse vectors. SAE initializations and final retrieval models are both released, with a blog post covering method, results, and ablations.
Related event: LightOn Open-Sources SlideOn Sparse Retrievers at 174M/339M, Claiming SOTA(23 posts)→
More from Models
- Dev Praises Opus 5.5's Great Intuition for Testing Its Own Code Changes — lucasmeijer · 2026-10-20
- US open-weights models jump in a week: Meta Muse Spark Open hits 38.9, Reflection's Beam 1.0 lands at 30.3 — ArtificialAnlys · 2026-10-20
- Ex-Google expert: data center capacity and electricity, not GPUs, are the real AI bottleneck — RihardJarc · 2026-10-20
- LLM citations vary run-to-run for the same prompt — so how valid are citation-overlap studies? — lilyraynyc · 2026-10-20
- ElevenLabs launches Eleven Realtime: speech-to-speech that talks in half a second and reasons — lukeharries · 2026-10-20
- nrehiew_: open-weight models are commodity-priced — don't even ask what OpenAI/Anthropic charge — nrehiew_ · 2026-10-20