RandSlot: Learning Compact Visual Document Representations with Random Soft Tokens
Dewen Guo, Shi Yu, Lingxiao Zhang, Yang Zhang, Tao XU, Dan Wang
cs.IR
2026-09-26
RandSlot trains 4 retrieval vectors from resampled random tokens. Qwen2.5-VL-3B scores 45.4 nDCG@5 on ViDoRe V3 vs CausalEmbed 42.6; zeros at inference keep it.
Visual document retrieval matches a text query to evidence spread across running text, tables, charts, and layout. A figure on a financial page often only makes sense with its row header and a footnote. Encoding pages as images keeps those relationships. ColPali-style methods turn a page into many vectors and let each query vector take its best matching page vector, then sum the scores (late interaction). Storage and compare cost scale with that many vectors: the full-token baseline keeps about five thousand per page.
The compression problem is joint quality. A few outputs still have to carry the match together. Light-ColPali and DocPruner merge or prune after encoding. MetaEmbed appends learnable tokens. CausalEmbed writes the model's latest vector back in as the next input. Those designs treat auxiliary inputs as parameters to fit or as carriers of content. RandSlot tests a cheaper option. A position with no preset meaning can still become a retrieval slot if it can attend to the content in front of it.
Ant Group and Tsinghua University share one causal encoder and emit K retrieval vectors per side, default K=4. After the content they append K-1 soft tokens (continuous placeholders): independent standard Gaussians, L2-normalized to unit length so a random vector cannot blow up attention scale. Draws are independent across examples, across query versus page, and across positions, and they are redrawn on every encoder call. No learnable token parameters are added.
Readout follows the causal prefix. The first vector is the last hidden state of the raw content. Each appended random vector creates a new last position and a new hidden state (a readout). Of the four outputs, one has seen the full page; the other three have also seen the earlier random slots. Outputs are normalized again, so a retrieval dot product is a cosine.
Scoring is directional MaxSim: each query vector takes the max cosine over the page's K vectors, then those maxima are summed. Training is in-batch query-to-document InfoNCE. Gradients update encoder adaptation weights only (LoRA rank 16). The random tokens have no parameters. The retrieval objective stays fixed while the auxiliary inputs keep changing, so the encoder cannot settle on a particular token value.
Pages are encoded offline into Kd vectors. At query time the encoder produces Kq vectors and ranks with MaxSim. The same checkpoint can keep sampling (RS-R) or put zeros in the extra positions (RS-Z). Positions and output count stay the same; no retraining.
Training uses the full ColPali split. Evaluation is page-level nDCG@5 on ten ViDoRe V1 tasks, four V2 tasks, and eight public V3 tasks. Backbones are Qwen2.5-VL-3B and Qwen3-VL-4B, four vectors each, widths 2048 and 2560. One epoch, 923 steps, eight A100 80GB GPUs, global batch 128.
Against published CausalEmbed (16 query vectors, 32 page vectors):
| Method | Budget | V3 nDCG@5 |
| CausalEmbed | 16/32 | 42.6 |
| RandSlot-3B | 4/4 | 45.4 |
| RandSlot-4B | 4/4 | 46.1 |
| Full-token Base | 5k | 50.2 |
Gains over CausalEmbed are 1.4 points on V2 and 3.6 on V1. RandSlot wins 20 of 22 tasks and all eight V3 tasks. Page vectors drop 87.5%. At 2048 dimensions and two bytes per scalar that is 16 KiB per page and 16 dot products per pair, versus 128 KiB and 512 dots for CausalEmbed.
Matched training recipe, matched four-vector budget, only the appended inputs change. V3 averages:
| Input strategy | 3B | 4B |
| Zero-Feedback | 43.9 | 45.1 |
| Fixed-Random | 43.8 | 45.1 |
| Learned-Feedback | 42.9 | 44.4 |
| Direct-Feedback | 42.1 | 45.4 |
| RandSlot | 45.4 | 46.1 |
On 3B with seeds 42, 123, and 2024, Zero-Feedback is 43.50±0.43 and RandSlot is 45.03±0.40, a 1.53-point mean gap. Learnable tables sit below fixed random on both backbones. On 3B, feeding outputs back in (42.1) sits below zeros. Resampling beats a frozen random draw by 1.6 and 1.0 points.
Swap the randomly trained encoder to zeros at inference: 3B stays at 45.4; 4B falls from 46.1 to 45.8. Both remain above models trained with zeros throughout (43.9 / 45.1). The gain lives in the weights. Serving does not need sampling.
Trained at four vectors and evaluated at other counts on 3B: zeros win at 1 and 2 vectors; RandSlot leads from 3 through 8. Six vectors reach 46.0 on V3; seven and eight stall at 45.9. Going from 4 to 6 raises query latency from 109.92 ms to 162.08 ms and drops page throughput from 4.01 to 3.29 pages/s.
Over 74,016 positive pairs, RandSlot spreads MaxSim winners most widely (marginal entropy H=0.909; distinct document slots per pair U=1.376), versus 0.789 / 1.256 for zeros. The best single document readout is weaker than the controls. Using the full set adds 6.34 points, versus 3.73 and 3.17 for those two controls. The four slots share the match.
Compact multi-vector visual document embeddings do not need a learnable token table or a loop that writes outputs back into the input. Training injects resampled unit vectors at fixed positions; late interaction puts the match into a few readouts. At serve time those positions can be zeros, so the index has no sampling noise.
The setting is narrow: screenshot pages, tight storage and compare budgets, about 16 KiB per page. Against CausalEmbed's 16/32 budget the representation is cheaper and V3 is 2.8 points higher. The paper is explicit that those ratios are representation cost, not end-to-end speedup.
The full-token baseline is still 50.2 on V3, so compact readouts have not closed the gap to patch-level matching. Four vectors is the default tradeoff; six add 0.6 points and nearly half again the query latency. This is a compact-readout tweak. Random inputs act as training noise.
All evidence is ViDoRe plus two backbones. Extra seeds cover only the 3B Zero-Feedback comparison. The larger backbone scores higher in absolute terms and gains less over zeros (4B +1.0, 3B +1.5). Pharmaceuticals and Fin-F drop relative to Zero-Feedback. Fin-E rises 2.0 points on 3B and falls 0.9 on 4B.
CausalEmbed numbers are published scores under the original recipe, not a rerun with shared hyperparameters. Direct-Feedback only matches budget and training procedure; on 3B it loses to zeros, so it does not stand in for CausalEmbed as published. MetaEmbed's Matryoshka multi-budget training is not reproduced here.
Training is locked at four slots. Changing K only at inference hurts at very small budgets. Code is planned; the paper does not ship a repo. The manuscript was drafted with substantial Codex help; the authors say the numbers come from their experimental records.