ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use
Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, Tat-Seng Chua
cs.CV, cs.HC, cs.MM
2025-04-04
ScreenSpot-Pro: 1,581 full-screen pro-app shots. OS-Atlas-7B leads at 18.9%; ScreenSeekeR, GPT-4o-guided search with no extra training, reaches 48.1%.
GUI agents still have to click. GUI grounding is that step: given an instruction and a screenshot, predict the pixel location of the target widget. Most existing tests sit on web pages, phone apps, or cropped patches. ScreenSpot, from SeeClick, is the last kind. Example crops land around 960×540, and the target occupies 2.01% of the image on average.
Professional tools do not look like that. Photoshop, AutoCAD, SolidWorks, MATLAB, and VS Code run above 1080p, often across two monitors. Menus are dense, icons are tiny, and extra documents sit on the same desktop. This paper, from NUS, East China Normal University, and Hong Kong Baptist University, rebuilds the test on full authentic screens. ScreenSpot-Pro holds 1,581 instructions on 1,581 unique high-resolution screenshots, spanning 23 apps, six genres, and Windows, macOS, and Linux. Targets cover 0.07% of the image, about 1/29 of ScreenSpot.
Higher resolution shrinks the relative box. Figure 2 already shows the same curve on ScreenSpot-v2: as the box gets smaller, SeeClick, OS-Atlas, UGround, and Qwen2-VL all drop. Extra tool windows on a professional desktop make the miss even easier.
The benchmark is the main artifact. Annotators had at least five years with the relevant app and worked through their normal routine. A silent capture tool overlays the current screen on a shortcut, so they draw the box and write the instruction in place instead of reconstructing the intent later. Screens are preferentially above 1080p with OS scaling off; dual-monitor setups are captured as one spanning image. Figure 1 puts 2560×1440 at 32.4% of samples. Two reviewers check each item, ambiguous instructions are rewritten to a single target, and boxes cover only the interactive region. A target is labeled icon only when no text hint is present; mixed icon-plus-label cases count as text. Every item also has a Chinese instruction, translated by GPT-4 and checked by bilingual authors.
The modeling observation is blunt: shrink the search region and accuracy rises. Three planner-free baselines share OS-Atlas-7B as the grounder.
ScreenSeekeR turns that shrink-the-search idea into a planner-guided visual search. GUI layouts are hierarchical: menus and property panes live in child panels. GPT-4o is the planner. Its own direct grounding is 0.8%, so it does not click. It names likely regions and nearby widgets ("New" often sits next to "Delete"). The grounder proposes boxes in those regions; small boxes are dilated, scored by how close a vote sits to the candidate center (σ=0.3), then suppressed with NMS and searched in score order. Direct grounding runs only when the crop is small enough (hyperparameter 1280 px), and the planner checks the box. No extra training.
End-to-end models collapse on these full professional screens.
| Model | Avg |
| OS-Atlas-7B | 18.9% |
| UGround-7B | 16.5% |
| Aria-UI (3.9B active) | 11.3% |
| ShowUI-2B / CogAgent-18B | 7.7% |
| SeeClick-7B | 1.1% |
| GPT-4o | 0.8% |
No other model clears 10%. OS-Atlas-7B is 28.1% on text and 4.0% on icons; Office is 27.4%, CAD 10.3%. Icons are hard because professional toolbars pack more functions than they can label, assume the user already knows the glyph, and those glyphs barely appear in web pretraining.
Chinese instructions make it worse. OS-Atlas-7B falls to 16.8%. UGround-7B falls from 16.5% to 7.7%.
Shrinking the search region helps immediately, still with OS-Atlas-7B:
| Method | Text | Icon | Avg |
| Direct | 28.1 | 4.0 | 18.9 |
| Iterative Zooming | 43.5 | 10.8 | 31.0 |
| Iterative Narrowing | 43.5 | 13.1 | 31.9 |
| ReGround | 55.7 | 15.1 | 40.2 |
| ScreenSeekeR | 64.1 | 22.4 | 48.1 |
Among planner-free methods, the simplest ReGround wins. ScreenSeekeR adds about 8 points to 48.1%, 2.54× the base model. Dropping recursive search and keeping only the first planner decision falls to 41.9%. Dropping neighbor inference is 46.4%. Majority vote instead of patch scoring is 46.8%. ScreenSeekeR reaches 64.3% on Office and 37.9% on CAD. Icons rise from 4.0% to 22.4%, still far below text at 64.1%.
Scores on cropped ScreenSpot patches overstate what a grounder can do on a real desktop. Professional software is full-screen, high-res, and the target is 0.07% of the pixels. A 7B specialist is under 20% out of the box. GPT-4o cannot click.
The transferable trick is search, not another fine-tune. One extra crop-and-reground step takes OS-Atlas-7B from 18.9% to 40.2%. ScreenSeekeR then shows that a planner which cannot ground can still cut the search space with layout knowledge. The cost is extra vision calls plus a GPT-4o API, so latency and spend both rise.
The benchmark matches real desktops; the method needs no extra training. Teams aiming at Photoshop, CAD, or MATLAB automation should measure here, not only on web and mobile numbers.
The authors restrict the suite to grounding, not planning or execution. Licensed professional software makes an OSWorld-style end-to-end environment legally messy. A correct click is not a finished job.
Even ScreenSeekeR is 48.1% average, 22.4% on icons, 37.9% on CAD. That is not ready to operate these tools. The planner is locked to GPT-4o; the ablations never swap in a weaker open planner, so it is unknown whether the 48.1% survives a cheaper stack. Under Chinese instructions the best model is 16.8%. Chinese workflows are mostly uncovered.
Whether 1,581 items cover the long tail of 23 apps, and how many samples are dual-monitor, is not broken out in the text. ReGround already hits 40.2%; the extra 8 points from ScreenSeekeR are not split by API cost versus extra crops.