UltraViT redesigns the vision encoder for on-device LVLMs and runs 1.7x faster
SamsungResearch · hf · 2026-07-29
UltraViT targets on-device LVLM latency with a new vision encoder
Samsung researchers introduce UltraViT, a vision encoder designed specifically for on-device large vision-language models (LVLMs), rather than treating the vision stack as a generic heavyweight backbone.
- The paper argues that prior compression work focuses too much on token reduction or smaller language models, while the vision encoder itself is often the overlooked bottleneck.
- UltraViT uses a pyramidal architecture that mixes heterogeneous spatial operators at the macro-block level, with the design guided by real device latency.
- For pretraining, the authors propose a two-stage generative strategy: first dense distillation to learn rich spatial features, then direct generative supervision from a frozen, capacity-mixed LLM.
- They report that this training recipe produces stronger semantic grounding for multimodal alignment than standard contrastive or SSL pretraining.
- Across experiments, UltraViT reaches a new SOTA for efficient LVLM encoding and runs on-device at about 1.7× the speed of prior encoder-centric baselines.
More from Research
- A KLS conjecture breakthrough links convex geometry, Monge–Ampère PDEs and ChatGPT — lihua_lei_stat · 2026-07-29
- Stream3D turns frozen 3D generators into streaming models with bounded memory — pliang279 · 2026-07-29
- Formal methods researchers discuss how to respond to AI progress at FLoC — swarat · 2026-07-29
- Hugging Face argues there is no truly tokenization-free LLM — cjmaddison · 2026-07-29
- PlayCanvas Update: Volumetric Fog Now Lit by Clustered Lights — willeastcott · 2026-07-29
- OKLS brings KL-optimal Shampoo to language model training with 1.45× parameter efficiency — aryaman2020 · 2026-07-29