EmbeddingGemma 2 finishing moves: single-trajectory checkpoint merging and int4 quantization-aware training
gabriberton · x · 2026-11-15
Thread [6/6] covers the last two training tricks for EmbeddingGemma 2:
- Model merging: like souping but with checkpoints of the same run — an unweighted average of the final finetuning checkpoint and a significantly earlier one from the same trajectory, mitigating overfitting degradation while keeping finetuning gains
- int4 quantization-aware training, with quantized versions released
The attached screenshot highlights "merge checkpoints along a single finetuning trajectory" from the paper.
More from Research
- ROS 2 Record-and-Replay for the SO-101 Arm Using MCAP and Foxglove — kamathsblog · 2026-11-15
- Ex-Meta researcher: internal code valsets without deliberate decontamination aren't valid tests — alexpolozov · 2026-11-15
- Google's new AIS team unveils DuoDiscover with 4.09x kernel speedups and LEAP formalizing Weil conjectures — VioletNPeng · 2026-11-15
- Compositional reasoning under RL post-training paper accepted to NeurIPS 2026 — yifeiwang77 · 2026-11-15
- How EmbeddingGemma 2 is trained: staged Gemini distillation down to 128-dim Matryoshka embeddings — gabriberton · 2026-11-15
- EmbeddingGemma 2 topped HuggingFace trending for a week, a spot usually held by LLMs — gabriberton · 2026-11-15