Just changing data representation swings multimodal models' scientific accuracy by 41 points

Ziyuan Wang · hf · 2026-11-13

A study tested whether multimodal models decide consistently on equivalent optical spectra representations:

Using measured spectra of silicon metasurfaces (MOCLIP) and gold nanoparticles, the same data was shown as four tables (nm, μm, scientific notation, eV) and two plots (nm, eV), evaluating d3-nano, Jev 1.13, three GPT-6 Luna configs, and Qwen3.5-2B.
Changing only the representation moved accuracy by up to 41 percentage points within one model, mostly on photon-energy tables/plots; asking in eV instead of nm helped some model-view pairs by 16-50 points and hurt others by up to 25.
With relevant data removed, unadapted models rarely answered "undetermined".
LoRA fine-tuning improved comparisons but its gains on unanswerable questions partly came from a shortcut — abstaining whenever input is truncated. Training with answerable truncations avoided the shortcut in all six seeds, with 90-100 point abstention gaps on held-out structures.
Conclusion: scientific multimodal models should be tested on equivalent representations and missing evidence, not just average accuracy; tasks, per-response results, and analysis code are released.

More from Research