Skild's robot foundation model S1: one .mp4 is the robot's .exe

Deepak Pathak's team at CMU demonstrated "physical prompt steerability" in Skild AI's robot foundation model S1: a single .mp4 video prompt acts as the robot's program, and the same weights switch tasks without retraining or fine-tuning — "a .mp4 is now the robot's .exe." Team member shikharbahl also shared the demo, stressing its out-of-distribution nature.

Confirmed

• In the same kitchen environment, four different video prompts drove S1 to execute four distinct behaviors, demonstrating prompt steerability.
• One task was far out-of-distribution — putting a plate into a toaster — which shikharbahl said he is 100% sure appears in no pretraining dataset (robot or human).
• S1 watched a human flip a pancake once and performed the same action directly on a real robot, though pancake flipping was not in its pretraining data.
• The same weights, with no fine-tuning or task-specific post-training, handled unseen tasks such as transplanting plants, pour-over coffee and kit assembly.
• Per the team's posts, scaling pretraining lets S1 acquire these capabilities at inference time — generalization comes from model scale rather than task-level training.

Unconfirmed

• Most posts attribute S1 to Skild AI (as does alexverem), but one post (m6) calls it a Physical Intelligence model; the attribution is confused in the repost chain, with Skild AI being the mainstream source.

Why it matters

• This marks a shift from programming/training per task toward an LLM-like "change the prompt, not the model" paradigm for robots — video as the robot's program.
• Success on out-of-distribution tasks suggests genuine generalization from the foundation model rather than replaying training data, a key advance for general-purpose robot foundation models.

2026-09-09 ~ 2026-09-10 · 7 related posts

• Physical Prompting Works: Robot Model S1 Follows 4 Different Video Prompts In One Kitchen — alex_verem · 2026-09-09
• [source] Video Prompts as Robot Programs: S1 Follows 4 Recipes in One Kitchen — deepakpathak · 2026-09-09
• [source] Robot S1 Steered by Video Prompts: Plate-in-Toaster Task Far Out of Distribution — deepakpathak · 2026-09-09
• Skild's S1 robot foundation model switches tasks via prompt, no retraining needed — rohanpaul_ai · 2026-09-09
• Skild's S1 learns pancake flipping from one video, no fine-tuning, as robot ICL emerges as a trend — deepakpathak · 2026-09-09
• Physical Intelligence shows S1 robot following video prompts, even putting a plate in a toaster — deepakpathak · 2026-09-10
• [source] S1 robot shows physical prompt steerability, infers novel OOD behaviors at inference time — deepakpathak · 2026-09-10