How SlideOn works: sparse autoencoder reconstructs token embeddings, then retrieval fine-tuning
tomaarsen · x · 2026-10-20
tomaarsen outlines the SlideOn/mSlideOn training recipe: SlideOn starts from DenseOn, mSlideOn from mLateOn.
A sparse autoencoder first reconstructs token embeddings using a few active features; the backbone and sparse encoder are then fine-tuned for retrieval, with base models already adept at retrieval helping. The diagram shows the full pipeline — tokenizer, encoder, projection, per-dimension max token pooling into a 32,768-dim sparse vector — with learned dimensions responding to recognizable concepts like "solar power" and "batteries".
Related event: LightOn Open-Sources SOTA Sparse Retrievers SlideOn and mSlideOn(24 posts)→
More from Research
- Community pushes Quasi-RH zero-free record to θ ≤ 0.874810282, Lean proofs pending — IgorCarron · 2026-10-21
- 3DRAE: Any 3D Scene in 1K Tokens — Diffusion in 3D Latent Space for Scene Generation — Dongxu Wei · 2026-10-21
- Deepak Pathak: There Is No Perfect Data Source for Robotics — chris_j_paxton · 2026-10-21
- AI models cluster into three personality families that cross lab boundaries, finds AA — ArtificialAnlys · 2026-10-21
- Discrete action representations match or beat continuous ones for robot policies — GuanyaShi · 2026-10-21
- ETSAM adapts Segment Anything 2 to treat 3D scans as video, scoring 0.812 on membrane segmentation — bravo_abad · 2026-10-21