GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding
Md Abrar Jahin, Md Rizwan Parvez
EMNLP 2026 Main Conference
cs.CL
2026-08-22
GUI-Primitives: 994 contrastive items, seven GUI spatial relations. Nineteen VLMs hit 32% point-in-box at best; 60-92% of clicks miss both candidates.
A computer-use agent has to ground first: look at a screenshot and turn "the button to the right of Save" or "inside the Layers panel" into a click. ScreenSpot and ScreenSpot-Pro only ask whether the click landed in the target box. OSWorld folds planning and execution into the same score. When the click is wrong, those benchmarks cannot say whether the model misread the relation or never found the two candidates.
GUI-Primitives splits the skill into seven elementary spatial relations and writes them as contrastive pairs. The screenshot and the anchor stay fixed; only the relation word changes; the correct target swaps between two designated candidates. A model that always clicks the most salient element scores one of the twins and misses the other, so the pair score is zero.
994 items, 497 pairs, seven primitives, 71 items per side. The two-candidate reference is 0.50 throughout. The seven split into three groups: left/right and above/below (cardinal); inside/outside (topology); same row or column, nearest/farthest, list ordinal, and visible/hidden (layout).
Two sources. The real arm has 290 desktop screenshots from UI-Vision (Web, macOS, VS Code, Office, professional apps) covering horizontal and vertical position, alignment, proximity, plus a 24-item list-ordinal slice mined from real menus. The synthetic arm has 704 procedurally rendered screens. Containment and occlusion exist only there, because real GUI corpora lack scalable parent-child or overlay metadata. In 253 pairs the verb and head noun also change, so the model cannot memorize the relation token alone.
Five annotators independently judged a stratified 196-item subset. Fleiss \(\kappa=0.94\) for well-formedness and \(\kappa=0.79\) for target choice. Eleven majority-invalid items were dropped, leaving a 185-item human-clean core on which annotators picked the right candidate 96.9% of the time. That is two-way selection accuracy, not the unconstrained click protocol the models face.
Nineteen vision-language models, greedy decoding. The primary metric matches ScreenSpot-Pro: whether the predicted point falls in the target box. Because models emit free coordinates, every prediction is also labeled target, distractor, neither, or invalid. Closed APIs downsample server-side, so images are resized to a known long side and coordinates are mapped back to the original pixels.
Three training-free interventions: Set-of-Mark (SoM) overlaid on the two designated candidates, so the model names a mark instead of regressing a pixel; a primitive-aware chain-of-thought prompt; and activation steering on residual layers 12 to 19 of Qwen2.5-VL-7B. The SoM condition is an oracle diagnostic. It supplies the candidate set the baseline task asks the model to find.
Claude Opus 4.7 leads at 31.3% on the full set and 32.4% on the clean core. GPT-5 is at 26.5% full. Qwen2.5-VL-7B is at 22.0% full and ties Claude Haiku 4.5 on the 196-item core (24.5% vs 25.0%). Humans sit at 96.9% versus Opus at 32.4%, a 65-point gap. The protocols are not matched, but the scale of the gap is.
Broken out by primitive, list-ordinal is the only one the top models have solved: Claude, GPT-5, and Qwen2.5-VL all land between 0.80 and 0.83. On left/right, Opus reaches 0.51, just on the reference line; everyone else is below. Best vertical score is GPT-5 at 0.32. On containment, occlusion, alignment, and proximity, every model's upper 95% bootstrap bound sits below 0.50.
Classify all 18,886 baseline predictions by landing region and the story changes. Between 60% and 92% of clicks fall outside both candidates. Conditional on landing in a candidate, target selection is 0.89 to 0.90 for horizontal, vertical, and proximity, and 0.548 / 0.554 for containment and occlusion, indistinguishable from 0.50. Most errors are candidate localization. Relation-word failure is confined to containment and occlusion. On real screenshots, 96.2% of clicks miss both boxes and strict point-in-box accuracy is essentially zero. On synthetic screens, loose accuracy reaches 0.60 to 0.76: many errors are a few pixels off.
GUI-Primitives tracks ScreenSpot-Pro at Spearman \(\rho=+0.74\) (\(p=0.015\), \(n=10\)). The sample is small; drop the strongest or weakest model and \(p\) rises to about 0.07.
SoM is the only intervention that moves the metric: GPT-5 30% to 87% (+57.1), OS-Atlas-Base-7B 10% to 52% (+42.1), Qwen2.5-VL-7B 22% to 57% (+35.1), Opus 31% to 40% (+9.2, Holm-adjusted \(p=0.08\), not significant). Chain-of-thought is +0.5 points and steering is +1.5, neither significant. Qwen already names the right relation in the CoT trace and still emits the wrong coordinate. On list-ordinal, where Qwen was already at 81%, switching to mark selection costs 32.4 points.
Shortcut controls behave. On Qwen2.5-VL-7B, a blank canvas drops 22.0% to 6.5%, a shuffled screenshot to 14.8%, and heavy Gaussian blur (\(\sigma=12\)) only to 16.7%, significant but under the pre-registered ten-point bar. Models lean on global layout more than on reading labels.
For anyone building a computer-use agent, the split is specific. On most primitives the model is not swapping left and right. It never lands on either candidate. Containment and occlusion are the cases where the relation word itself carries no signal. Chain-of-thought and residual steering do not fix this. Handing the model a candidate set (a detector, SoM, a crop) can lift selection by tens of points. That is an upper bound, not a deployable method, and detector errors pass through.
The ScreenSpot-Pro correlation says the benchmark is not a toy. Scale does not close it: the strongest closed models sit between 24% and 31%. A 7B Qwen2.5-VL ties Haiku.
Containment and occlusion live only in the synthetic arm, so the "no relation-word signal" claim does not travel. After model fixed effects, per-primitive logistic coefficients are at most 0.07 in absolute value; only the model-level association is claimed. Downstream evaluation stops at ScreenSpot-Pro, not OSWorld task success. Version 1 is English, static, click-only. SoM marks at most two candidates; a crowded real UI may not repeat the 35 to 57 point jump. Point-in-box is binary; anchor collapse and direction inversion show up only as qualitative cases. Llama-3.2-11B-Vision emits a default coordinate on about 42% of items (1.2% accuracy) and inflates the off-candidate rate.
There is also a left/right prior: clicks fall in a candidate region about 2.7 times more often when the target is the left element, and within-region accuracy is 0.972 for "left of" versus 0.717 for "right of". The paper treats this as a frequency-prior hypothesis, not a causal result.