Trainable [DUMP] prefix tokens reduce representational interference in retrieval embeddings
_reachsumit · x · 2026-11-06
A CIKM 2026 paper proposes prepending trainable [DUMP] tokens during embedding training to absorb residual token-level noise (e.g., attention-sink artifacts) before pooling. It keeps architecture, pooling and loss unchanged, optimizing the tokens via the standard contrastive objective, and consistently beats finetuned baselines across encoder and decoder embedding models on retrieval and long-context benchmarks.
More from Research
- UWaterloo 3D prints Invar36 flexures with directly mounted interferometer optics — jwt0625 · 2026-11-06
- AMD paper: a single generalized MLP upscales textures on-the-fly at render time, saving VRAM and disk — ssh4net · 2026-11-06
- OpenAI researcher invokes 2019 double descent paper in scaling-law spat — willdepue · 2026-11-06
- Encord debuts REACH egocentric hand-tracking model, cutting errors up to 70% — Scobleizer · 2026-11-06
- Only 16% of AI benchmarks measure organizational difficulties of real jobs — ben_burtenshaw · 2026-11-06
- Skillware framework claims 97.3% cost cut as 3B-8B local models beat frontier LLMs — RossPeili · 2026-11-06