Gaze Target Estimation Anywhere with Concepts
Xu Cao, Houze Yang, Vipin Gunda, Zhongyi Zhou, Tianyu Xu, Adarsh Kowdle, Inki Kim, James M. Rehg
CVPR 2026 Code and Benchmark
cs.CV, cs.AI
2026-08-12
Name a subject by text or click, and GazeAnywhere predicts where they look end-to-end. It runs 12x faster than the best 3B pipeline and halves error on an OOD clinical set.
Gaze target estimation answers a simple question: where is the person in this image looking? In the wild, the answer is fragile. The dominant recipe is a multi-stage pipeline: detect the people in the scene, draw a head or face bounding box around the subject, then estimate gaze from the cropped head region. Every stage demands explicit inputs (head box, pose, depth), and a single miss cascades downstream. Crowded scenes, poor lighting, and children, whose faces are hard to detect to begin with, trigger these failures most.
A less-discussed gap: you cannot name the subject in natural language. Recent end-to-end, bottom-up models emit everyone's gaze in one pass, but they lack identity association. They can tell you where each person looks, not where "the boy in the red shirt" looks. Pinning gaze to a specific person still falls back to an external detector plus post-processing, and the cascaded error returns. This paper closes that gap with PGE (Promptable Gaze Target Estimation): specify the subject by text or visual prompt, then estimate gaze end-to-end.
GazeAnywhere, the first model built for PGE, resolves subject localization, in/out-of-frame presence, and gaze in a single forward pass. Prompts come in two forms: text, organized into appearance, position, pose, and action ("the boy in the red shirt"), or visual, a coordinate like [0.52, 0.48].
The architecture keeps both the image encoder (a DINOv3-L ViT) and the text encoder (dino.txt) frozen; only the projection layers, a stack of transformer detector blocks, and three decoder heads are trained. Text tokens, image patch tokens, a head token (for localization), and a presence token (for in/out judgment) are concatenated into one sequence and fused by the detector. The three heads output a 64x64 gaze heatmap (conv upsampling, BCE against a Gaussian target with sigma=3), a head box (an FFN regressing [x,y,w,h], trained with L1 plus GIoU), and an in/out binary classifier (focal loss), all under a joint objective.
Two choices deserve explanation. First, why freeze the encoders. Ablations show that unfreezing either the image or text encoder hurts clearly; DINOv3's pretrained features are robust enough here, and fine-tuning invites overfitting and feature drift. Second, why bind head localization into the same gradient path as gaze. The ablation shows that adding the head loss improves both gaze estimation and presence prediction; coupling "find the head" with "estimate where it looks" is exactly what dissolves the cascaded error of two-stage pipelines.
The authors built Gaze-Co, a dataset of 120K prompt-annotated image pairs (69.6% from GazeFollow, 19.6% from VideoAttentionTarget, 10.8% from ChildPlay), with concept phrases generated by Gemini 2.5 Pro and human spot-checks keeping the error rate at or below 1%. Four test sets follow.
Main results (GazeAnywhere-DINOv3-L, 870M params):
| Test set | Metric | GazeAnywhere | Strongest two-stage baseline |
| GazeFollow-Concept | AUC | 0.958 | 0.954 |
| VAT-Concept | AP | 0.879 | 0.861 |
| ChildPlay-Concept | AP | 0.906 | 0.914 |
| Child-SC (OOD) | L2 | 0.090 | 0.172 |
The strongest two-stage baseline pairs Gaze-LLE with RexSeek, a 3B detector. Two details matter more than the table. Speed: GazeAnywhere runs 96 ms per image against 1183 ms for the 3B baseline, about 12x faster, and the CLIP-L variant drops to 35 ms. It also matches the oracle: feeding the ground-truth head box to Gaze-LLE caps AUC at 0.961, while GazeAnywhere reaches 0.958 with no box given. The biggest lead is on Child-SC, the out-of-domain clinical set of child social-communication videos: L2 error 0.090 versus 0.172, roughly halved.
Honesty compels a caveat: it does not dominate every metric. On ChildPlay AP the 3B baseline edges it (0.914 versus 0.906), though GazeAnywhere wins on L2 there (0.098 versus 0.119), and the GazeFollow AUC margin is just 0.004. The clean wins cluster in speed, OOD generalization, and matching the oracle. General-purpose VLMs do far worse at gaze outright: Qwen3-VL-8B and Gemini 2.5 Flash reach only 0.65 and 0.66 AP on VAT, with roughly double the L2 error.
The authors also wire the model into an MLLM as GazeAnywhere Agent, running real-time social-gaze analysis on DigiLens ARGO glasses; per-minute gaze-shift MAE falls from 9.2 with a raw MLLM to 2.3.
Gaze estimation is, for the first time, freed from the engineering burden of supplying a head box. A user names a subject by sentence or click, and the deployment bar drops sharply. Going end-to-end removes the cascaded-error source exactly where today's open-vocabulary detectors are weakest: the strongest OVD, OWLv2, hits only 70% accuracy on child head and face detection in Child-SC. The natural homes are AR for real-time social analysis and developmental screening (autism-spectrum-related social-communication assessment), where private data cannot leave the device, making a local model a requirement rather than a nicety.
The authors concede that text prompts remain intrinsically ambiguous and need structured categories to constrain them, and that private sets like Child-SC cannot pass through cloud MLLMs, so local inference is mandatory. Two points invite scrutiny. First, GazeAnywhere still leans on the frozen image encoder's ability to recognize people; the paper does not stress-test what happens when subjects are effectively unrecognizable (heavy occlusion, extreme distance) and whether the end-to-end path collapses too. Second, the margins on GazeFollow and VAT are thin, and part of the SOTA claim rests on Child-SC, a private set that releases metrics but not data, which makes independent reproduction hard. The paper positions itself as a step toward clinical use, not a finished clinical tool, and that framing is honest.