S1 robotics model criticized for misleading 66% success metric
johncoogan · x · 2026-08-30
Rob Toews critiques the S1 model, highlighting two key limitations:
- Training Specificity: S1 was explicitly trained for in-context learning, making demos less significant.
- Misleading Metrics: The 66% success rate refers to per-step success, not task completion. For a 20-step task, the actual success rate is 0.02% (0.66^20), meaning success likely occurs only once in 4,000 tries.
The post challenges optimistic narratives about robotics foundation models, urging a distinction between prediction loss and actual task success.
More from Embodied
- Chinese robots hold 86% of global shipments as Nvidia extends CUDA playbook — rohanpaul_ai · 2026-08-30
- Musk: Affordable Level 5 Self-Driving Requires AGI, Still 5-10 Years Away — elonmusk · 2026-08-30
- Tesla Expands Dallas Robotaxi Service Area to 80 Square Miles — elonmusk · 2026-08-30
- Robot simulation created within 24 hours of release, dev stunned by speed — Kyrannio · 2026-08-30
- Tesla AI Hiring: Building On-Edge Inference Stacks for Robotics Prototypes — yunta_tsai · 2026-08-30
- FIFA-style robot soccer game built with open-source RL policies — gaganghotra_ · 2026-08-30