Encoder-free multimodal LLMs re-learn encoder-like traits in early layers
rdesh26 · x · 2026-11-14
A JSALT-originated study finds encoder-free multimodal LLMs (Gemma-4-12B, Inkling-Small) re-learn encoder-like properties in early layers: representations become invariant to cross-modal context and resemble self-supervised encoder features. This motivates "in-fusion" — folding encoder layers into the LLM backbone while reserving them for a modality. Ablations at identical 0.4B params/depth: widening shared FFNs scores 40.1, adding a branch for all tokens 42.3, making the branch vision-only 45.0 — beating a standalone encoder's 44.4, showing the vision-reserved branch is the key factor.
More from Multimodal
- Speechify's Simba 3.5 tops voice arena at $6.58/1M chars, 6x cheaper than Eleven v4 — ArtificialAnlys · 2026-11-14
- Multiple #OpenSourceHiggsfield winners vanish from GitHub with 404s — Clair_Personality · 2026-11-14
- Uncensored Qwen-Image-2.1 quantized model trends on Hugging Face — chatrlabs · 2026-11-14
- MiniMax H3 Image editing tested at 60MP: concept art to realistic photo in ~2 min on RTX 4090 — daroamer · 2026-11-14
- Maket launches Custom Materials: generate floor and wall textures from text or images — testingcatalog · 2026-11-14
- User shows off stunning Gemini-generated Milky Way hydrogen simulation — intellectronica · 2026-11-14