JSALT Insights: Multimodal LLMs Should Drop Modality-Specific Encoders

rdesh26 · x · 2026-08-08

Developer rdesh26 shared two key architectural insights on multimodal LLMs after attending the Omnimodal Encoders presentation at JSALT 2026:

Drop modality-specific encoders: As model backbones scale up, separate encoders become redundant. It's more effective to feed raw features (like STFT or image patches) directly to the LLM, following the "bitter lesson." Models like Gemma-4-12B prove that the initial layers of no-encoder models spontaneously learn encoder-like representations.
Towards "Agentic Audio": For complex agentic workflows, instead of relying on a massive general-purpose multimodal model, it's better to use small, task-specific experts. An orchestrator can write code to call these tools sequentially to solve multi-step multimodal tasks.

More from coding & agent