LightOn and H Release NeoMME, a Single-Tower Native Multimodal Multilingual Encoder

The LightOn team (Hcompany) released NeoMME, a family of natively multimodal, multilingual encoders with 260M and 800M parameters. According to author antoinechaffin, NeoMME departs from the common approach of using large generative VLMs as representation models; instead, it trains a native encoder from scratch that handles both text and images, is multilingual, is designed for speed, and has no separate vision tower.

Confirmed

• Architecturally, it uses a single bidirectional Transformer that jointly processes multilingual text tokens and raw image patches, requiring no pretrained vision tower, text encoder, or decoder, and supports a 16384-token context.
• The pretraining objective is a masked discrete diffusion text objective, trained from scratch.
• Two sizes are provided, 260M and 800M, both released on Hugging Face.
• The 260M model outperforms peers on the ViDoRe v3 retrieval benchmark, leading the sub-800M parameter tier.
• The team has also done downstream fine-tuning work based on the encoder.

Why it matters

• NeoMME represents a technical route distinct from "first train a generative VLM, then extract representations": training an encoder as the goal from scratch, with small parameter counts and fast inference, well suited to latency-sensitive scenarios like multilingual multimodal retrieval.
• The 260M model leading its size class and even larger models on ViDoRe v3 suggests that native multimodal pretraining with a diffusion objective can be competitive at small representation-model scales, offering a new option for lightweight multimodal retrieval.

2026-09-03 ~ 2026-09-03 · 5 related posts

• [source] NeoMME: single-tower multimodal encoder leads sub-800M models on ViDoRe v3 visual retrieval — _reachsumit · 2026-09-03
• [source] Hcompany Releases NeoMME: Multimodal-Native Multilingual Encoder with Masked Discrete Diffusion — Hcompany · 2026-09-03
• NeoMME: single-tower multimodal multilingual encoders beat sub-800M rivals on ViDoRe v3 — CShorten30 · 2026-09-03
• [source] LightOn ships NeoMME: natively multimodal encoders at 260M/800M with no vision tower — antoine_chaffin · 2026-09-03
• NeoMME: From-Scratch Multimodal Encoders at 260M/800M With No Vision Tower — CShorten30 · 2026-09-03