LLM-powered revival of Put-That-There brings speech and gesture window control to XR
twi_mar · x · 2026-09-07
- A new arXiv paper (2511.02378) revives Bolt's classic "Put-That-There" for modern headsets: an LLM fuses a semantically segmented 3D environment, live app metadata, and users' speech, pointing, and head-gaze cues to emit JSON window-placement actions.
- It supports explicit commands ("Place Google Maps on the coffee table"), deictic speech plus gestures, and high-level goals ("I need to send a message"), letting the LLM infer which apps to open and how to lay them out — intent-driven interaction without manual window juggling.
- The post quotes OpenAI's GPT-6 Astra launch, an agent that can do anything you can do on a computer, signaling this context-aware interaction direction.
More from coding & agent
- Astra sparks AI-CAD wave: creators rebuild landmarks and engines across Blender, SOLIDWORKS, Fusion — burhop · 2026-09-07
- OpenAI internal data: over 80% of successful 32-hour agent tasks still needed human intervention — DataLearnerAI · 2026-09-07
- threepointone: UX is about depth, agent experience is about breadth — threepointone · 2026-09-07
- dhh migrated a whole site to Astro with Muse Code for just 20 cents — alexandr_wang · 2026-09-07
- Microsoft dev blog: building AX evals that actually work, when scalar scores mislead — lee_stott · 2026-09-07
- UMass AutoIndex turns chunking into code an LLM writes, with hypothesis-level validation lift — CShorten30 · 2026-09-07