Inside EmbeddingGemma 2's Training: Gemini Distillation and INT4 Quantization
EmbeddingGemma 2 is trained via staged distillation from Gemini Embedding 2, starting with text-only alignment and compressing embeddings down to 128 dimensions. Final tricks include single-trajectory checkpoint averaging and INT4 quantization-aware training.
2026-11-15 ~ 2026-11-15 · 2 related posts
- How EmbeddingGemma 2 is trained: staged Gemini distillation down to 128-dim Matryoshka embeddings — gabriberton · 2026-11-15
- EmbeddingGemma 2 finishing moves: single-trajectory checkpoint merging and int4 quantization-aware training — gabriberton · 2026-11-15