Training-Free Speech-Centric Omni Understanding with Frozen VLMs
Ankan Deria, Hanoona Rasheed, Xilin He, Fahad Shahbaz Khan, Salman Khan
eess.AS, cs.CV, cs.SD
2026-08-08
TFO feeds timestamped Whisper transcripts into a frozen VLM. On 56 benchmarks and 21 languages it matches native omni on speech and better keeps vision and reasoning.
Native omni models add a dedicated audio encoder to a VLM and pay for audio-video-text alignment. A new backbone means retraining the audio path, and the alignment step can wash out vision, reasoning, and domain skills the original VLM already had. Audio is dense and noisy, and it has to lock onto visual events; native models still miss sounds or invent them from the picture.
The missing control is simple. Does every new VLM need native omni training? Can speech-centric omni capability be added while the backbone stays frozen? Where do richer acoustic features still matter?
TFO (Training-Free Omni) is a plug-in that turns a frozen VLM into a speech-centric omni system. No architecture change, no parameter update, no multimodal re-alignment. Audio is routed as language. The visual pathway is untouched.
Whisper sits outside the VLM and splits audio into segments, each with text, language, start/end times, and confidence. Only segments with confidence ≥ 0.65 are kept; if none pass, audio context is omitted so silence and hallucinated transcripts do not enter the prompt. When the task needs temporal reasoning, timestamps stay in the prompt so spoken spans can be aligned with visual events. Fusion is ordinary prompting: filtered transcript, system instruction, visual input, user query. Optional spoken replies go through CosyVoice3; that TTS path is never used to build evaluation audio.
The hypothesis is narrow. On many audio-visual tasks, speech mainly supplies linguistic evidence, and the VLM already knows how to reason over language-conditioned vision. TFO measures how far that hypothesis goes.
Matched comparisons cover four families and five settings: Qwen2.5 3B/7B, MiniCPM4.5, NVILA/OmniVinci, and Qwen3-30B-A3B. The suite is 56 benchmarks, plus CoVoST2 in 21 languages.
On nine audio-visual averages, Qwen2.5-3B moves from 43.2 to 46.2 (+3.0), 7B from 45.8 to 48.0 (+2.2), VILA +0.4; MiniCPM4.5 drops 2.2 and Qwen3 drops 0.8. The largest gains sit on speech-conditioned temporal video: WorldSense +14.5 on Qwen2.5-7B. Nine audio-only averages rise in every setting (+1.5 / +1.4 / +4.0 / +13.5 / +1.4). VILA goes 50.3 to 63.8, with VoiceBench +51.3. CoVoST2 averages rise +7.9 / +13.5 / +18.4 / +11.1 / +6.8, MiniCPM4.5 from 45.6 to 64.0. Every variant drops 1.5–9.5 on MMAR-Bench; music and environmental sound are the transcript blind spot.
Freezing also keeps the backbone. Image averages are higher in all five settings (+1.8 to +2.4). Video is higher in four (+2.3 to +3.8). Every TFO variant beats its omni counterpart on the three coding sets. Medical averages rise in all five (+0.5 to +1.7). AVHBench draws the boundary: TFO is weaker when the model must verify that a sound is actually in the audio; it wins across four families when it must verify that an object implied by audio is visible, because the visual path was never touched.
Ablations: the frozen VLM alone already beats Qwen2.5-Omni on WorldSense and Video-Holmes. Adding Whisper beats Omni-3B on five of six AV sets. Dropping timestamps costs 2.3 on AVUT-Gemini and 1.4 on Daily-Omni.
A new VLM does not have to wait for backbone-specific audio alignment to become usable as a speech-centric omni model. Training cost moves to inference: total latency on AVMeme rises from about 0.7–2.4 s for native omni to 1.3–3.1 s for TFO, and Whisper can be amortized when several questions share one clip. The multilingual lift shows that a strong ASR front-end can hand language coverage to a frozen VLM.
That is not an argument to stop omni training. Non-speech acoustics, emotion, and audio-visual matching still want acoustic features. TFO is a control and a shortcut: route speech first, train a dedicated path where acoustics are the evidence, and keep watching whether alignment erodes the backbone.
The representational ceiling is built in. Transcripts discard timbre, environmental sound, and musical structure. Appending textual predictions from MELLOW or SenseVoice does not stably beat Whisper alone; acoustic evidence does not come back by being written down. Sequential Whisper adds latency, so real-time full duplex is out of scope. The matched comparisons are tied to existing omni checkpoints; they do not say what a different ASR or fusion template would do. MiniCPM4.5 and Qwen3 lose slightly on the AV average, so language routing is not uniformly better.