NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference
Aurélien Lac, Tony Wu
cs.IR, cs.AI, cs.CV
2026-08-31
NeoMME is a from-scratch 260M/800M single-tower encoder; the retriever scores 0.523/0.556 nDCG@10 on ViDoRe v3 and indexes 2048px pages at ~2x ColModernVBERT throughput.
Visual document retrievers often start from a generative VLM: a pretrained vision tower plus a causal language model, then used as an encoder. The generative stack’s parameters and compute come along for a non-generative job. ColPali-style systems embed page screenshots for late interaction. ModernVBERT swaps the decoder for a bidirectional encoder but keeps a SigLIP2 tower. Dual towers and “vision encoder + LM” give the two modalities different compute paths and split the training and serving lifecycle.
NeoMME’s bet is that retrieval should use a bidirectional encoder trained for representations, with raw patches and text sharing every Transformer layer from the first block.
Both the 260M and 800M models train from random init. Text uses an ALBERT-style factorized embedding (256-d lookup, then a projection). Images are split into 32×32 RGB patches and mapped with LayerNorm plus a 2-layer MLP. There is no pretrained ViT and no patch merging. Context is 16,384 tokens, enough for about two 4K pages at 32-pixel patching. Most layers use symmetric sliding-window attention (half-windows alternate 256 and 1024); every sixth layer and the last layer are global. The stack adds 2D RoPE, GQA, and QK-Norm.
Pretraining is masked discrete diffusion on text. Loss is only at masked positions. Image patches stay visible and condition the denoiser. Text-only segments sample a mask rate from U(0,1); multimodal segments sample from U(0.30,1) so the model cannot lean on leftover words. Planned data is 524B packed tokens, 55% text and 45% multimodal, for 500k steps.
Retrieval fine-tuning trains two heads on one backbone: a 128-d late-interaction head scored with MeanMaxSim, and a Matryoshka dense head. One forward pass produces both. Fused MaxSim kernels (LIK) drop the peak at 4,096 document tokens from 672MB to 193MB.
ViDoRe v3 is reported on its eight public tasks. Late-interaction nDCG@10 is 0.523 (260M) and 0.556 (800M). The 260M model leads every evaluated system strictly below 800M, and beats the best other <300M model by 26.1 / 11.5 / 5.4 nDCG points on v3 / v2 / v1. 0.523 is within 0.2 points of 3.75B ColQwen2.5 at about 1/14.4 the size. The 800M model is 3.2 points above ColQwen2.5 and within 0.9 points of 850M Vultron Flash.
Against ColModernVBERT: 0.5505 vs 0.4364 on v2, 0.8666 vs 0.8177 on v1. On BEIR-15, late-interaction beats dense by 18.3 points (260M) and 14.4 points (800M); the 800M late-interaction score is 0.513. On an L40S at 2048×2048, 260M indexes 51.3 pages/s versus ColModernVBERT’s 26.0, about 1.97×. Hierarchical token pooling (factor 8) plus asymmetric quantization (int8 queries, binary documents) compresses the 260M multi-vector from 1.5MB to 6kB per page (255×) while keeping more than 95% of baseline nDCG@10.
At 90% text masking, visible page patches raise masked-token accuracy by 38.4 points (260M) and 40.5 points (800M), so pretraining is using the image.
For visual RAG, this is a page encoder that does not carry a VLM decoder, released Apache 2.0 with a Transformers implementation. The 260M model moves both quality and indexing throughput in the small-model band, and 255× compression makes late-interaction storage plausible for large page corpora. It is not a general vision tower: frozen 16-shot natural-image probes average 13.2 accuracy. Document transfer is the comfortable setting (RVL-CDIP fine-tune on 6,000 examples reaches 81.5).
Text-only pretraining is 290B tokens against ModernBERT’s 2T. There is no image reconstruction or image-level contrastive loss, which likely explains the weak frozen natural-image probes. Retrieval supervision is far smaller than mLateOn (430k text queries vs 660M contrastive pairs), so the BEIR gap is not identified as architecture versus data. Mixed-modality corpora are in the training mixture but UniDoc-Bench and MixBench were not run. There is no matched ablation of architecture, data, and objective, so “dropping the vision tower” is not an isolated cause. Visual retrieval language coverage is narrow.