Omni-LLMs Know More Than They Say: NAVER's OmniCatalyst Adds 7.29% via Self-Generated Text
naver-ai · hf · 2026-10-14
A KAIST/NAVER AI Lab paper uncovers a "knows-but-cannot-say" gap in omni-modal LLMs: models often perceive information from vision, audio, and IMU inputs correctly but fail to express it in responses.
- They propose Token-level Modality Alignment (TMA), measuring alignment between per-modality representations and response tokens
- Non-textual representations are less aligned with and less attended to than textual ones, explaining uneven modality preferences
- OmniCatalyst, a perceive→infer strategy that complements inputs with self-generated text, improves performance by +7.29% across seven Omni-LLMs with more balanced modality preferences
- Knowledge-extraction phases increase TMA, suggesting better alignment makes non-textual knowledge more accessible during generation
More from Multimodal
- Another Beta Tester Ports a three.js Game to Kaigen 3D — gdechichi · 2026-10-14
- HKU and Tencent open-source Multics, a multi-player world model that trains on 4 players but scales to 10 — Scobleizer · 2026-10-14
- MiniMax Design Agent V3 launches with 30% lower creation costs and asset reuse — Hailuo_AI · 2026-10-14
- Meta-Video shows frame sampling, backend and temperature skew video understanding benchmarks — HildeKuehne · 2026-10-14
- QwenImage 2.1 shines as an identity-preserving edit model in Reddit workflow writeup — Cautious_Chicken_604 · 2026-10-14
- Seedance 2.5 demo: singing forest elf with tiny creatures, prompt included — azed_ai · 2026-10-14