FLINT: Fixed-budget Late Interaction
Himank Handa, Negar Arabzadeh, Matei Zaharia
cs.IR, cs.CL
2026-10-20
FLINT pools each query and document to 32 vectors, running 1.5–2.8× faster end-to-end than ConstBERT on long queries, with indexes 5.3–9.3× smaller than ColBERTv2.
ColBERT keeps one vector per query token and per document token, then scores with MaxSim: each query vector takes its maximum dot product against the document vectors, and those maxima are summed. At a similar model scale this usually ranks better than a single-vector bi-encoder. The index grows with document length, and the number of comparisons grows when the query gets long.
ColBERTv2 cuts storage with residual compression. PLAID filters candidates through cluster centroids. Both still store a vector per token. ConstBERT pools every document to a fixed 32 vectors, so storage no longer tracks document length, and ranking on MS MARCO and BEIR holds up. The query side is untouched. That is cheap on BEIR, where queries are mostly short keywords, a handful to a few dozen tokens. It gets expensive on CRUMB's multi-constraint questions, on the vague spoken descriptions in TREC Tip-of-the-Tongue 2025, and on BrowseComp+ evidence finding. Mean query length there runs from about 100 tokens to about 230, and ConstBERT's search cost is tied to query length again.
FLINT locks the vector count on both the document and the query, then checks how much ranking quality is left on long, out-of-distribution queries.
A shared transformer encodes the query and the document separately into variable-length contextual vectors. A cross-attention pooler, adapted from the Perceiver Resampler, sits in between. Each side has V learned latent vectors. The latents are the attention queries; token states are keys and values. Every latent reads the whole sequence and writes one output vector. Longer text changes what those vectors contain, not how many there are. A position-wise feed-forward block then expands the dimension, applies GELU, and projects back.
The backbone, the pooler weights, and the final linear map are shared. The only side-specific parameters are the two latent sets, Lq and Ld. Stacking self-attention after the pooler made document vectors collapse into near-duplicates and the scores collapsed with them, so the block stays at one cross-attention layer plus one feed-forward layer.
Scoring is still MaxSim, now over V vectors on each side. For every query vector, take the max dot product with the V document vectors, then sum. The comparison count is V×V, independent of input length. The default V is 32, the same document budget ConstBERT uses, so a storage gap cannot be explained by storing more document vectors. V can also be 16 or 64. Training data matches the two baselines: MS MARCO. The backbones are BERT and ModernBERT-large.
The metric is NDCG@10. On short-query BEIR, BERT FLINT averages 0.452, behind ConstBERT at 0.469 and ColBERTv2 at 0.502. A few sets go the other way. On Quora, FLINT reaches 0.848, above ConstBERT at 0.817 and close to ColBERTv2 at 0.855. With ModernBERT, V=32 averages 0.530, against ConstBERT 0.518, ColBERTv2 0.534, and a single-vector baseline at 0.449. On ordinary retrieval, a fixed budget at best ties token-level late interaction.
The long-query sets are where ranking separates. With BERT, FLINT's CRUMB average is 0.145, above ConstBERT at 0.136 and ColBERTv2 at 0.143. TREC-ToT 2025 moves from 0.047 to 0.065, also above ColBERTv2 at 0.046. BrowseComp+ is 0.063, against 0.055 and 0.057. With ModernBERT, FLINT beats ConstBERT on all three complex suites. On TREC-ToT 2025, the longest and most conversational queries, ConstBERT scores 0.070, under half of FLINT's 0.147. Against ColBERTv2 the split is mixed: CRUMB is 0.205 to 0.184 in FLINT's favor; TREC-ToT is 0.147 to 0.155 and BrowseComp+ is 0.105 to 0.135. The single-vector model scores 0.042 and 0.022 on those last two.
The budget trades off directly. Dropping ModernBERT FLINT from 32 to 16 hurts every suite, but it still beats the single-vector model everywhere and beats ConstBERT on the three complex suites, at about half the index size. Raising V to 64 barely moves BEIR (0.531 vs 0.530) and keeps helping complex queries: TREC-ToT reaches 0.164 and BrowseComp+ 0.108.
On the long-query corpora, end-to-end mean response time is 1.5–2.8× faster than ConstBERT. The widest gap is TREC-ToT 2025, about 6.4 million documents and a mean query length of 204 tokens: BERT drops from 74.3 ms to 26.1 ms, ModernBERT from 140.9 ms to 51.3 ms. Against ColBERTv2 the same corpora are 10.5–36.2× faster with BERT and 23–170× faster with ModernBERT. The 170× figure is that ModernBERT cell, where ColBERTv2 takes 8707.9 ms end to end. On NFCorpus, where queries average 7 tokens, the two response times are nearly tied.
Almost all of the gap is search. On TREC-ToT with BERT, encoding grows by 0.005 ms per extra query token for both models. Search grows by 0.004 ms per token for FLINT and 0.31 ms per token for ConstBERT. Search itself is 2.5–4.8× faster. PLAID does not score the whole corpus. Query vectors probe nearby centroids, and only those candidates get MaxSim. ConstBERT scores 2.3–4.6× as many candidates and compares each one with 5–7× as many query vectors.
On disk, BERT FLINT and ConstBERT are almost the same size, since both store 32 vectors per document. Against ColBERTv2 the FLINT index is 5.3–5.8× smaller with BERT and 6.8–9.3× smaller with ModernBERT. On TREC-ToT the ModernBERT index falls from 70G to 7.5G. Single-vector indexes are smaller still on the three large corpora, and their search takes about 1 ms. V=64 costs 2.8–4.9× the search time of V=32.
| Setting | Baseline | FLINT |
| BEIR mean NDCG@10, BERT | ColBERTv2 0.502 / ConstBERT 0.469 | 0.452 |
| BEIR mean, ModernBERT | ColBERTv2 0.534 / ConstBERT 0.518 | 0.530 |
| TREC-ToT 2025, ModernBERT | ConstBERT 0.070 / ColBERTv2 0.155 | 0.147 |
| Same setup, end-to-end latency | ConstBERT 140.9 ms / ColBERTv2 8707.9 ms | 51.3 ms |
| Index vs ColBERTv2 | one vector per token | 5.3–9.3× smaller |
Earlier late-interaction speedups either compress token vectors or shorten only the document. The query stayed one vector per token. Once questions get long, ConstBERT's search time steepens with length. FLINT flattens that part: fewer candidates, and MaxSim per candidate goes from query-tokens × 32 to 32×32.
Index size stays in ConstBERT's band, 5.3–9.3× below ColBERTv2, and it does not grow with document length. The change is where pooling sits. The training objective is unchanged, so the pooler can drop into a multi-vector stack that already runs PLAID. Timing for both systems uses PLAID, and the reported speedups do not add a separate dimensionality reduction on top. V of 16, 32, or 64 means about half the index, the default budget, or 2.8–4.9× the search time in exchange for higher scores on complex queries. Swapping BERT for ModernBERT leaves those relative gains in place.
On BEIR with BERT, the average really is behind both baselines: 0.452 versus 0.469 versus 0.502. With queries of only a handful to a few dozen tokens, a fixed 32 query vectors do not buy extra accuracy. The abstract's comparable is closer to the ModernBERT result, 0.530 versus 0.534.
The 170× latency ratio is the largest corpus and the longest queries, measured against ColBERTv2. Versus the fixed-budget baseline, end-to-end speedup is 1.5–2.8× and the index is barely smaller. The 5.3–9.3× storage cut is against token-level models, and 9.3× is ModernBERT on TREC-ToT. That storage gap belongs to ColBERTv2.
Absolute scores on the complex suites stay low. BrowseComp+ at 0.105 beats ConstBERT at 0.067 and still trails ColBERTv2 at 0.135. V=64 reaches 0.164 on TREC-ToT while search costs 2.8–4.9× more. The main tables skip XTR, WARP, and MUVERA. The comparison is held to one backbone and one MS MARCO training setup, which keeps it clean and keeps it an architecture comparison. Public ModernBERT multi-vector checkpoints were missing, so both baselines were retrained on the same schedule, and that variance is not reported on its own.
NDCG@10 holds up with a smaller candidate pool. Recall@1000 appears only in the appendix, and the main text does not say how much recall was cut. Pooled vectors on BERT are more concentrated in direction. Whether probing fewer centroids drops relevant documents is not tested directly. Training is MS MARCO only, so the long-query gains are zero-shot. Whether V=32 is still enough after training on long queries is left open.