OVEarth-Bench: Evaluating Category Breadth and Query Diversity for Open-Vocabulary Earth Observation
Kaiyu Li, Zepeng Xin, Zixuan Jiang, Jing Fu, Lanxuan Xue, Lingyu Zhang, Xiangyong Cao
cs.CV
2026-07-29
A new remote-sensing benchmark with 172 hierarchical categories and 3,134 vocabulary, referring, and reasoning queries tests open-vocabulary Earth observation; 49 zero-shot models top out near 42% IoU, and general MLLMs beat domain-specific ones.
Open-vocabulary Earth observation (EO) asks a model to localize a geographic feature from a sentence, not from a fixed label set it was trained on.
Existing benchmarks do not really test this. Most reuse conventional remote-sensing datasets, so category coverage is narrow and query forms are limited. Some test only vocabulary phrases, some only referring expressions, almost none cover vocabulary, referring, and reasoning at once. A model that tops such a benchmark may have memorized a small category set rather than acquired open-vocabulary skill. OVEarth-Bench closes that gap on two axes: broader categories, including fine-grained, long-tail, and negative cases, and more diverse queries.
The benchmark is built on freshly collected EO imagery with 172 hierarchical categories spanning land cover, transport, industrial facilities, public services, and agriculture. Three query types map to three abilities.
Scale: 1,346 vocabulary phrases (425 positive, 1,029 negative), 732 referring expressions, 1,056 reasoning queries, 590 human-verified masks over 520 images and 4,810 boxes, 91.8% of them under 1% of image area. Queries are not hand-written; a two-stage LLM pipeline generates phrases, referring expressions, reasoning prompts, and negative candidates from the category name, image, and mask, and human reviewers confirm negatives are truly absent and each query uniquely grounds the target. All 49 model variants run under a unified zero-shot protocol covering mask and box localization (horizontal and oriented boxes).
The headline numbers are cold. The best mask ma-IoU is 38.75% on vocabulary, 41.32% on referring, and 37.96% on reasoning, all from Rex-Omni+SAM. The best box [email protected] sits at only 24.53%, 35.56%, and 26.12%.
| Query type | Best ma-IoU | Best model |
| Vocabulary | 38.75% | Rex-Omni+SAM |
| Referring | 41.32% | Rex-Omni+SAM |
| Reasoning | 37.96% | Rex-Omni+SAM |
Three findings matter more than the absolute scores. First, MLLM-based methods dominate: nine of the top ten on vocabulary, all ten on referring and reasoning. Second, EO-specific models gain little; only one cracks the vocabulary top ten, none make the referring or reasoning top ten, and the best EO-specific result trails the maximum by 7 to 15 IoU points. Third, rankings collapse on narrow category sets: sampling five categories gives a median Spearman correlation of 0.84 against the full ranking and retains only three of the top five; twenty categories lift it to 0.95 and recover four of five; fifty are needed to reach 0.98.
One more is easy to miss: accurate localization does not imply reliable rejection. A model that masks positive targets well is only weakly correlated with correctly judging that a category is absent, so hallucinated localization is undercounted.
For remote-sensing practitioners the practical takeaway is sharp: if you benchmark on small category sets, your ranking may be noise. Rankings stabilize only as category coverage widens.
That general MLLMs beat EO-specific models points one way. The breadth of foundation-model pretraining is worth more than domain fine-tuning here, and building a stronger open-vocabulary EO model likely means combining the semantic breadth of a general MLLM with EO-specific spatial competence (multi-scale high-resolution encoding, orientation-aware decoding) rather than tuning harder on narrow data. And "is this concept present at all" should be trained as a separate, calibrated objective, not assumed to fall out of localization.
The authors concede several. The findings are descriptive, not causal: training data, scale, output interfaces, and inference budgets differ across models, so whether MLLMs win because of paradigm or because they are larger and newer cannot be separated. The dataset is small, 590 masks, and some categories have only one or two samples, making category-level results fragile. Detection comparison uses mostly horizontal boxes because few methods emit oriented boxes, so orientation-aware localization is left undecided, and evaluation focuses on category level rather than taxonomy granularity.
The biggest doubt matches theirs. Rex-Omni, X2SAM, and SAMTok are simultaneously the largest and newest models in the field, so "MLLMs win" confounds scale with paradigm. This benchmark alone cannot split them; controlled ablations would.