LLaViT: giving the LLM a bigger role in visual understanding in multimodal models
burkov · x · 2026-11-01
The article argues a common flaw in multimodal LLM design: treating image features as if already compatible with language-model inputs, when they start in a very different representation space that the LLM itself must progressively interpret. It tests this via LLaViT, a LLaVA modification with three changes: separate attention projections for image vs text tokens, unrestricted bidirectional attention among image tokens, and image features drawn from multiple depths of the vision encoder — giving the LLM a more active role in visual understanding.
More from Research
- DIY UBI math with real data: about $2,000 per person after 30 years, not a living wage — edmrunmachine · 2026-11-01
- GRASP authors compare with SuperMimic: extracting more optimization signal from sim experience — hankyang94 · 2026-11-01
- Harvard's GRASP refines human demos in physics sim for dexterous hands and Unitree G1 — hankyang94 · 2026-11-01
- DeepMind researchers publish book offering Bayesian framework for assessing AI consciousness — mhutter42 · 2026-11-01
- Graphical Algebraic Geometry Paper Links Polynomial Reasoning to Quantum ZH Calculus, Accepted at LICS 2026 — ctjlewis · 2026-11-01
- Is There a Modality-Agnostic Taxonomy of ML Techniques? Reddit Asks Which Patterns Repeat Across Text, Image and Audio — OkCampaign8400 · 2026-11-01