UltraVBERT: a 247M-parameter vision-language encoder for visual document retrieval
kalyan_kpl · x · 2026-10-28
UltraVBERT is a 247M-parameter vision-language encoder designed as a base for visual document retrievers. It combines the UltraBERT-150M text encoder with the SigLIP2-B/16-512 vision encoder, producing contextual embeddings for every text and image token.
Architecture: builds on ModernVBERT — swaps in UltraBERT as the text encoder, normalizes connector output, raises the page limit from 2,048px/16 tiles to 3,072px/27 tiles, and adds a training stage on rendered text pages.
Training: masked language modeling on answers in image-grounded conversations (80% <|mask|>, 10% random, 10% unchanged), with DoRA adapters (rank 64, alpha 16) on both encoders and the MLM head at peak LR 1e-4; the connector is trained in full.
Three models are already available: UltraVBERT (base encoder), BiUltraVBERT (single-vector retriever), and ColUltraVBERT (late-interaction retriever, with demo).
More from Research
- Intent-first Text2Cypher: LLM never writes Cypher, but only 1 of 4 runs correct — JeremyCMorgan · 2026-10-28
- Playcanvas proposes SOG v3: 4-band SH, opacity above 1, per-gaussian blind data — willeastcott · 2026-10-28
- DecisionFormer open-sources a toolkit to convert any LLM into a decision model — foolishpixel · 2026-10-28
- Singapore launches S$150M MARVEL-AI materials discovery platform with Cusp AI — jonkhler · 2026-10-28
- New tutorial traces AI's evolution from LLMs to "thinking systems" via Test-Time Scaling, Loop, Harness, RSI — rsasaki0109 · 2026-10-28
- Anders Sandberg on a wild 48 hours: group classification formalized, Kakeya, RCE, AI incident — anderssandberg · 2026-10-28