How EmbeddingGemma 2 is trained: staged Gemini distillation down to 128-dim Matryoshka embeddings
gabriberton · x · 2026-11-15
Thread [5/6]: EmbeddingGemma 2's training mostly relies on distilling Gemini Embedding 2 in stages:
- Text-only distillation first
- Multimodal distillation with images, videos, audio, and documents
- Post-distillation stage using a contrastive loss plus a spread-out loss to distribute embeddings evenly across feature space, adding Matryoshka sizes from 768 down to 128
The quoted context explains the motivation for Matryoshka Representation Learning (Google Research, 2024): embeddings are costly to store and retrieve for huge databases (e.g., 100B documents), and nested representations let one vector serve multiple dimension budgets.
More from Research
- Google's new AIS team unveils DuoDiscover with 4.09x kernel speedups and LEAP formalizing Weil conjectures — VioletNPeng · 2026-11-15
- Compositional reasoning under RL post-training paper accepted to NeurIPS 2026 — yifeiwang77 · 2026-11-15
- EmbeddingGemma 2 topped HuggingFace trending for a week, a spot usually held by LLMs — gabriberton · 2026-11-15
- SJTU's Value-as-Memory Gives Robot Policies Task-Progress Memory, +29.4pp on Real Tasks — jiqizhixin · 2026-11-15
- Genesis Team Launches Eden, a Robot Learning Framework That Builds Tasks 1.6-4x Faster Than Isaac Lab — zhou_xian_ · 2026-11-15
- University of Bonn Reverses Aging Markers in Human Cells by Up to 60 Years — Dr_Singularity · 2026-11-15