Trainable [DUMP] Prefix Tokens Reduce Representational Interference in Retrieval Embeddings
nlpai-lab · hf · 2026-11-06
Researchers from Korea University and Soongsil University propose a minimal method for improving retrieval embeddings: prepending trainable [DUMP] tokens during embedding training, optimized via the standard contrastive objective. The tokens absorb residual token-level signals—such as attention-sink artifacts and recurrent positional patterns—that otherwise interfere with the pooled representation.
The approach leaves architecture, pooling strategy, and loss unchanged, yet improves results on standard retrieval and long-context benchmarks, offering a low-cost, plug-and-play enhancement for embedding models.
Related event: Trainable [DUMP] Prefix Tokens Clean Up Retrieval Embeddings(2 posts)→
More from Research
- Reviewer scandal: Pangram flags 3 of 4 peer reviews as 100% AI-generated on a rejected paper — nsaphra · 2026-11-06
- AI2's ToyBlocks: Sim-Trained Robot Stacks Words on Real Hardware, 15/30 Attempts — allen_ai · 2026-11-06
- AI2 Open-Sources ToyBlocks, a Simulated Tabletop for Controlled Robot Learning — allen_ai · 2026-11-06
- Do hidden states encode beyond next-token info? A researcher answers antirez — burny_tech · 2026-11-06
- UMich research finds generative AI uses up to 180x more energy on non-English text — radamihalcea · 2026-11-06
- Open-source decision model family Drex debuts, Drex Max tops decision benchmarks — arankomatsuzaki · 2026-11-06