Technion's MIST stress test finds irrelevant images shift 20% of VLM judge labels regardless of content

Technion · hf · 2026-10-01

Technion researchers introduce MIST (Misleading-Image Stress Test): 200 English sentences readable figuratively or literally, paired with an aligned image, a misleading one, or none, with guidelines requiring the label to be decided from the sentence alone.

Key findings:

Across 13 VLM judges, an aligned image flipped 20.5% of labels and a misleading one 19.4% — nearly identical, and both above the 11.6% caused by simply deleting the ignore-the-image instruction.
Only 37% of flipped labels moved toward the sense the image actually depicted; agreement with human annotators was unchanged whether images were absent, aligned, or misleading.
What moves a judge is that an image is present, not which one it is — so substitutability verdicts for VLM-as-annotator describe the evaluation configuration as much as the model itself.

Original post →

Technion's MIST stress test finds irrelevant images shift 20% of VLM judge labels regardless of content

More from Models