PORTAL: training-free projector init skips multimodal alignment with 135x fewer FLOPs
anshulkundaje · x · 2026-10-14
A NeurIPS 2026 paper dissects what projector training actually does in multimodal LLMs and proposes PORTAL as a replacement.
- Finding: projector training mainly provides coarse alignment
- PORTAL is a training-free projector initialization—no paired captions, no gradient updates—letting you skip the projector stage entirely and go straight to multimodal instruction tuning
- Across 16 benchmarks: beats the standard 558k-pair pipeline in 3 of 7 model configs, and averages 98% of its score in the other four
- With light projector training, it matches the baseline with 135x fewer alignment FLOPs
Paper, webpage, and code are all public.
More from Research
- DatologyAI launches Datology Labs, an open research initiative for the science of data — josh_wills · 2026-10-15
- DatologyAI launches Datology Labs for open collaborative data science research — pratyushmaini · 2026-10-15
- Prime Intellect experiments show ECHO beats pure RL in turn-capped, no-thinking robotics-like environments — DimitrisPapail · 2026-10-15
- Incoming Berkeley professor lists open questions: pre/post-training alignment, safe agents — canondetortugas · 2026-10-15
- Berkeley's QF3: off-policy flow RL with filtered Q-gradients trains humanoids from scratch — berkeley_ai · 2026-10-15
- VLMs May Turn Clearly Superhuman in Vision Within a Year or Two, Researcher Predicts — arnabdotorg · 2026-10-15