GPT-Policy teaches robots from human video demos via in-context learning, no retraining
KyeGomezB · x · 2026-09-19
The paper "In-Context Robot Learning with VLM Agents" introduces GPT-Policy, a framework that turns robot learning into prompting:
- Core idea: a frozen commercial VLM (e.g. GPT-6 Astra) adapts to new tasks purely via in-context learning—no gradient updates or task-specific parameter changes.
- Architecture: a context compiler preserving task-relevant visual transitions, a VLM proposing robot-tool actions, and a constrained controller that verifies and executes each action and reports outcomes.
- Key findings: on real robots, human video demonstrations (without robot action labels) improve task completion; aligned action references yield further gains on contact-sensitive tasks.
- Significance: brings few-shot learning into the physical world—teaching a robot a new behavior could look like showing it an example.
Authors span Shanghai Innovation Institute, HUST, FDU and other institutions.
Related event: GPT-Policy: Robots Learn New Tasks via VLM In-Context Learning(2 posts)→
More from Embodied
- DISCO distilled tree search named CoRL 2026 best-paper finalist, top 6 of 687 — yining_hong · 2026-11-04
- Berkeley's Value Forcing: enforcing Bellman consistency at every noise level for accurate RL value functions — berkeley_ai · 2026-11-04
- 22 Creative SAM 3.1 Use Cases From Meta's AI Developer Hackathon, From Wildfire Detection to Document Reassembly — Vjeux · 2026-11-04
- Ex-OpenAI robotics intern breaks down viral Astra robot-painting demo on Cerebras podcast — AccBalanced · 2026-11-04
- Embodied Agent Arena: Closed-Weight VLMs Lead All Five Robot Task Domains, Open-Weight Models Lag — Haojian Huang · 2026-11-04
- PrimeBot's PrimeSim: a generalist world model simulator trained on cross-embodiment demos — Travis Davies · 2026-11-04