Reasoning Quality Matters: Combating Reasoning Collapse in LLM-based Embedding Learning
Zihan Gong, Xiaohan Ye, Jiangchao Yao, Jinsong Lan, Xiaoyong Zhu, Xu Chen
cs.IR
2026-09-17
Alibaba CoFree uses reference-guided SFT and dual-reward RL to keep embedding reasoning retrieval-faithful. CoFree-4B gains 2.8 nDCG@10 over Qwen3-Embedding-4B on 22 datasets.
Turning an LLM into a dense encoder usually takes one of two routes: compress the text directly, or first generate a short "reasoning" string and embed the concatenation. The second route sounds right and often fails. Generation collapse: empty strings, runaway repetition, incoherent text. Semantic collapse: fluent prose that is useless for retrieval, such as mixing coral reefs with Coral Reef Casino because the surface tokens match. The paper calls both failures reasoning collapse. Collapsed reasoning pollutes the vector. Sometimes no reasoning is better.
The Taobao retrieval group at Alibaba asks two operational questions: how to restore generation without wrecking a pretrained embedding geometry, and how to make the restored text actually separate positives from hard negatives.
CoFree starts from an existing LLM embedding model and trains in two stages. CoFree-4B is initialized from Qwen3-Embedding-4B; CoFree-1.5B from gte-Qwen2-1.5B-instruct.
Stage one, reasoning restoration, writes the reasoning then reads the <eos> hidden state as the embedding. Three losses run together: causal LM to produce the text; InfoNCE with positives, hard negatives, and in-batch queries; and a reference-guided MSE that pulls the reasoning-conditioned vector toward a frozen copy of the backbone embedding of the raw input, so the geometry does not forget. Those terms, weighted, are the SFT objective.
Stage two is GRPO with dual rewards. The embedding reward is the cosine margin between the positive and the negative after the sampled query reasoning. The reasoning reward uses a separate Qwen3-Reranker-4B to score query-plus-reasoning against document-plus-reasoning, again as a positive-negative margin. The margin, not a raw score, blocks a cheap hack: generic padding that lifts both sides equally. GRPO's KL penalty limits generation drift; an extra embedding-space MSE pins policy vectors to the post-SFT reference.
Data: about 7.0M raw query-document pairs are filtered into RTED, 3.6M tuples with teacher reasoning from Qwen3-235B-A22B. SFT imitates that text. RL resamples reasoning from the original tuples.
Across 22 datasets (10 MTEB English retrieval plus 12 BRIGHT), overall nDCG@10:
| Model | Overall | MTEB | BRIGHT |
| Qwen3-Embedding-4B | 38.0 | 64.1 | 16.3 |
| Contrastive FT-4B | 32.6 | 50.9 | 17.3 |
| CoFree-4B | 40.8 | 66.4 | 19.4 |
| stella 1.5B | 31.4 | 52.2 | 14.0 |
| CoFree-1.5B | 34.6 | 58.5 | 14.7 |
Contrastive fine-tuning on the same pairs yields 32.6 at 4B. CoFree SFT alone already reaches 39.9; RL adds 0.9 to 40.8. Removing the reference-guided loss drops SFT from 39.9 to 37.1. Degeneration rates judged by three LLMs fall from about 100% before SFT to under 4% after; seq-rep-4 falls from 99.37% to 3.32%. On the 1.5B model, mean pairwise accuracy of three held-out rerankers rises from 51.30% after SFT to 64.00% after full RL, and stalls at 50.90% if the reasoning reward is removed.
In an e-commerce search A/B test, CoFree ran as an extra recall channel: CTR +0.02 and CVR +0.12 absolute, orders +13.56% and GMV +8.15% relative, fully deployed since January 2026. CoFree-1.5B's end-to-end throughput is 4.4-5.4x Search-R3-Small, averaging 39.4 generated tokens on MTEB.
Think-then-embed is already in TTE, Search-R3, and UME-R1. This paper writes down the failure mode clearly: generation is not the same as retrieval-useful generation, and it can hurt. Reference anchoring plus dual rewards is a patch you can put on Qwen3-Embedding. SFT carries most of the score; RL mainly repairs reasoning quality. For a team already serving Qwen3-Embedding and willing to pay for a short generation, this is a followable increment. RTED is slated for public release, which lowers the reproduction bar.
BRIGHT-Code is the exception: CoFree-4B falls from 11.5 to 9.9, and code-oriented supervision is thin.
The authors flag the training bill: two stages, teacher generation, reward-model scoring; about 24 hours of SFT and 48 hours of RL for the 4B model on 32 GPUs. The code-retrieval drop is not given a causal study, only the observation that CodeSearchNet and CoSQA together are about 5.5% of retained tuples. The online CTR/CVR figures are absolute 0.02 / 0.12 without the baseline denominator; the relative order and GMV numbers are the ones to read. Reasoning-quality judges are stated to be distinct from the training reranker, but it is still models grading models. Transfer tests on frozen encoders show the reasoning text itself helps; they do not show that every new backbone will keep the same gain after fine-tuning.