S1 robotics model criticized for misleading 66% success metric
johncoogan · x · 2026-08-30
Rob Toews critiques the S1 model, highlighting two key limitations:
- Training Specificity: S1 was explicitly trained for in-context learning, making demos less significant.
- Misleading Metrics: The 66% success rate refers to per-step success, not task completion. For a 20-step task, the actual success rate is 0.02% (0.66^20), meaning success likely occurs only once in 4,000 tries.
The post challenges optimistic narratives about robotics foundation models, urging a distinction between prediction loss and actual task success.
More from Embodied
- How Microsoft optimized Windows for Nvidia's Arm-based RTX Spark laptops — The Verge AI · 2026-10-15
- John Deere's second-gen autonomous tillage: 16 cross-checking cameras beat sensor drift — lukas_m_ziegler · 2026-10-15
- Europe's Application-Layer Bet: Can the Rotterdam Humanoid Center Deliver? — ingliguori · 2026-10-15
- VC thesis 'Data at the Edge': cheap sensors plus AI are rebuilding the internet's data flywheel offline — rebeccakaden · 2026-10-15
- PingpongDuo shows humanoid robots playing doubles table tennis — arankomatsuzaki · 2026-10-15
- Robotics data gap: just 300K hours of manipulation data globally vs 100M+ needed, $3B spend projected — 0xSammy · 2026-10-15