Revisiting put-that-there, context aware window interactions via LLMs
Riccardo Bovo, Daniele Giunchi, Pasquale Cascarano, Eric J. Gonzalez, Mar Gonzalez-Franco
cs.HC
2025-11-04
An unfine-tuned GPT-4 fuses Quest scene semantics, window metadata, and pointing/gaze into JSON place/remove actions for XR; no user study is reported.
Bolt's 1980 Put-That-There showed that speech plus pointing can drive underspecified spatial commands. Consumer headsets now reconstruct head and hands in real time and label walls and furniture. The XR "infinite desktop" makes display area almost free. Placing the windows is still manual. Pavanatto and colleagues have already measured the cost: organizing windows across a panoramic virtual display is cognitively heavy.
This paper, from Imperial, Birmingham, Bologna, and Google, wires an LLM to that sensor stack and reruns Put-That-There. The goal is not a better pointer. It is to let a user issue a full command, say "put that there," or name only a task, and have 2D windows snap onto real planar surfaces.
There is no fine-tuning. GPT-4 (Chat Completions, March 2025 snapshot) consumes three JSON streams and emits executable actions.
Scene understanding uses the Meta Quest Scene API, which auto-labels about ten classes: wall, floor, cabinet, bed, chair, sofa, table, door, window, bookshelf. That set is too coarse, so users add about thirty manual labels, including counter, desk, whiteboard, refrigerator, and sink. Planar regions are extracted from the semantic mesh: adjacent faces are grouped by normal dot product, PCA estimates orientation, and each plane inherits its semantic tag plus a visibility score. Windows come from WindowMirror, an open-source mirror of existing PC windows into XR. Each window has an id, size, current surface (or none), and name.
User behavior includes speech, pointing hovers (object id plus hover duration), and head direction. Very short hovers are dropped as noise. When pointing is used, the window's current surface is excluded as a target, so the model does not "place it back where it is." The system prompt locks the assistant to JSON: a response field and an actions array of [place|remove, windowid, surface] triples. The prompt states the task, the meaning of each field, worked examples, and rules for deixis: prefer visible surfaces, ignore short hovers, avoid redundant placements.
Three interaction styles share that pipeline. A full command: "Place Google Maps on the coffee table." Speech plus pointing: "Put that there." A goal only: "I need to send a message," after which the model picks a chat app and a currently visible surface. A high-level goal can emit several actions at once, for example placing Maps, Notes, and Calendar for trip planning. The model can also add tools by functional relation: finding images for a slide deck opens a browser and a slide editor together.
This is a systems paper. There is no user study, and no accuracy, completion-time, or NASA-TLX number. The paper does not compare against manual window management or a rule-based placer. What can be checked is the capability envelope, not effect size.
Implementation choices that can be checked: the action space is only place and remove; windows are resized to the surface automatically; semantics are ten Quest classes plus about thirty manual labels; surfaces carry visibility (0.8 for a cabinet in the example); pointing events carry hover time (1.5 s in the example); the model is GPT-4 with no fine-tune. The authors frame three shifts: one user act can emit many placements; explicit coordinates give way to goals; functional links among apps are inferred on the fly instead of being written as rules.
XR productivity has been stuck on cheap screens and expensive layout. This paper gives a reproducible engineering path: pack scene semantics, window metadata, pointing, and head pose into JSON, and let an off-the-shelf LLM emit actions, with no special-purpose window-management model. For spatial-agent builders, the contract is the part to copy: action triples, visibility, hover-noise filtering, exclusion of the current surface.
One-to-many and goal-centric input move control. If the user says "I need to send a message," the system chooses the window and the surface with no confirmation step. The paper flags trust, transparency, and feedback as open. That is the right flag. Automation saves the hands. It may move uncertainty into "why did it put that there."
The authors list a user study on cognitive load, task performance, and experience as the next step. Until that study exists, there is no evidence that goal-centric placement is easier than explicit manipulation, and no measurement of mis-placement, multi-user use, or long sessions. Semantic granularity depends on hand labels; the automatic ten-class set cannot ground "coffee table," which will not scale. Plane finding is coplanarity plus PCA, so curved surfaces and cluttered desks will be dirty. Actions are place and remove only: no resize, stacking, undo, or confidence. GPT-4 is a black box, layouts are not reproducible, and latency is unreported. The stack is Quest Scene API plus WindowMirror; there is no Vision Pro counterpart. Examples are living-room offices, not factories, clinics, or multi-person meetings.