REIGN: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling
Devrim Çavuşoğlu, Emre Akbaş
EMNLP 2026
cs.CL, cs.AI, cs.IR
2026-08-31
REIGN trains a small encoder on cached chunk vectors, cutting training cost ~10,000x. 55M hits 67.31 nDCG@10 on GoodWiki-Long and ties 1.6-4.3x larger models on patents.
Dense retrieval over long documents still pays token-level attention: sequence length squared blows up fine-tuning. Current long-context embedders reach 8K to 32K by stretching RoPE, sparsifying attention, or turning a billion-parameter LLM into an encoder. Document-to-document contrastive learning also lacks paired long positives and realistic hard negatives. Short-text MTEB does not supply that supervision.
REIGN treats a pretrained text embedder such as GTE or E5 as a frozen Guidance Network. Documents are windowed at the GN context size, typically 512 tokens, and GN outputs are cached in HDF5. The trainable piece is a small Transformer with no positional encoding, a permutation-equivariant set function over the chunk-embedding sequence, then average pooling. Single-chunk inputs stay with the bare GN: with N=1 there is nothing to aggregate, and on short MTEB tasks REIGN trails the GN by 5 to 7 nDCG@10.
Training uses a three-way cosine loss. Positives are GPT-4o-mini rephrasals of long Wikipedia articles, topical distractors are partial matches at λ=0.5, and other in-batch items are negatives. The release includes GoodWiki-Long-Synthetic: 17,854 long queries and 53,562 corpus docs, query-side median about 5,817 tokens. The default encoder is base-l3 at about 22M; stacked on a GN the total runs from 55M to 357M.
On GoodWiki-Long, sparse methods lead because rephrased positives still overlap lexically: BM25 scores 76.82 nDCG@10. Among dense systems, 55M REIGN+GTE-small reaches 67.31, edging 568M BGE-M3 at 67.21, and beating Jina-v3 at 63.93 and Stella-1.5B at 63.66 by 3.4 to 3.7 points, also beating chunked GTE-large at 65.94. Chunked mean-pool already adds 1.7 to 2.7 over truncation; the cross-chunk encoder adds another 0.7 to 2.2, largest at the small GN. Scaling the GN from 33M to 335M does not lift REIGN's in-distribution nDCG@10.
Zero-shot on LoCo, chunked GTE-small already hits 68.20 macro nDCG@10, above BGE-Large 56.5, Ada-002 63.2, and Jina-v2 67.2. REIGN on that GN reaches 68.92 at 55M. REIGN+GTE-large at 357M reaches 70.77, 0.65 behind E5-Mistral at roughly 20× the size. Bare chunked GTE-large is already 71.38; REIGN drops 0.38 and 0.61 on GTE-base and GTE-large. Gains concentrate on subtasks that need evidence across chunks. Legal term-overlap tasks are weak for both.
On DAPFAM patent retrieval, GoodWiki-only REIGN+GTE-large scores 33.10 nDCG@100 versus 32.43 for chunked GTE-large. Against 572M Jina-v3 and 1.54B Stella the paired test cannot separate them. Cross-IPC positives stall near 5 for every system, a dataset contrast problem. Extra DAPFAM fine-tuning does not beat zero-shot: the best run moves 32.68 to 32.73.
With cached GN vectors, queries take 0.40 to 0.52 ms on an RTX 4090, versus 19.8 to 118.2 ms if the GN runs per query. Per-document training cost is reported as about four orders of magnitude below chunked Transformer fine-tuning.
When the unit of retrieval is a whole document and evidence is spread across chunks, a 22M encoder on frozen chunk vectors is a cheaper lever than widening an LLM window. A 55M stack already matches dense models an order of magnitude larger on synthetic long wiki and on patents. In production it should sit beside the original GN, not replace it on short queries. LoCo is the task condition: once chunked mean-pool saturates query-to-passage subtasks, the extra encoder can cost points.
The claim is parameter and training efficiency, not raw accuracy. GoodWiki-Long positives are rephrased distillations, so BM25 still wins on lexical overlap. The DAPFAM edge over Jina and Stella is not significant, and versus chunked GTE-large REIGN is the larger model. The 5 to 7 point short-text drop is structural. Legal long text stores the signal in term overlap, which pooled embeddings discard. A new GN forces a full encoder retrain and a new cache. The positional-encoding ablation also used a different training recipe than the released models.