Token pooling combines well with embedding quantization and Matryoshka
antoine_chaffin · x · 2026-11-12
sentence-transformers maintainer tomaarsen shares results showing that token pooling works remarkably well in combination with (asymmetric) embedding quantization, and even with Matryoshka representation learning — though Matryoshka is lossier, it suits models with native dimensionality well above 128.
antoinechaffin adds that depending on backbone size, pooling to 64 tokens works without much loss for larger backbones, but not for smaller ones.
More from Research
- Teknium: Hermes memory research shows ~15% better generalization than finetuning — Teknium · 2026-11-12
- Teknium on Unify's memory research: Hermes shows both memorization and ~15% gain over finetuning — Teknium · 2026-11-12
- Robotics take: get intelligence first, then distill to 27B sparse MoE — ChongZzZhang · 2026-11-12
- Meta releases MetaEncoder: one bi-encoder for retrieval and decision-making via natural language — pmttyji · 2026-11-12
- Physics-informed learning by IITM professor slashes AI simulation compute costs — ravi_iitm · 2026-11-12
- InvisibleMentor: AI That Watches Your Screen Recording to Suggest Better Workflows — burkov · 2026-11-12