Prior contextual embedding training treats supporting-context chunks as negatives
antoine_chaffin · x · 2026-11-03
antoinechaffin identifies the core training flaw of prior contextual embedding models: they assume one "gold chunk" per query and treat every other chunk as a negative.
But supporting context is not a distractor — a chunk defining an entity, resolving a reference, or giving a relevant condition may be necessary to interpret the answer. Treating it as a negative corrupts the training signal.
Related event: Distilling Graded Chunk Relevance Fixes Contextual Embedding Training(2 posts)→
More from Research
- New "private subquadratic" effort draws skepticism: just add weights to a Transformer — teortaxesTex · 2026-11-03
- GPT-6 Astra editing Qwen3.5 weights for 6x math gains? Skeptics call it cherry-picked demos — teortaxesTex · 2026-11-03
- OSVerify catches proof-killing errors that GPT-6-Pro and Codex both missed — james_y_zou · 2026-11-03
- Stanford and NUS hearables amplify hidden sound textures in real time at UIST 2026 — HanGuo97 · 2026-11-03
- Three LLM agents share one Godot escape room: logging what agents say vs. what they do — ViceGodIm · 2026-11-03
- Steering Qwen-Image-2.1 with Qwen3-VL's internal states, no training needed — zacharynado · 2026-11-03