Per-task robotics eval breakdown suggests top 2 labs already training on robot tasks
teortaxesTex · x · 2026-10-26
In a discussion thread, @teortaxesTex shares a per-task breakdown of a robotics evaluation distribution and draws two notable conclusions:
- The top 2 labs are clearly already training on robotic-relevant tasks — the results show a step-function pattern (e.g., on "Pick box", Sol 6 performs as poorly as 1017/Kimi, while Sol 6.1 makes clear progress).
- Per-task variance is dramatic: 1017 is the only model scoring 0% on "Drawer open"; in 4 of 9 tasks both 1017 and K3.1 score 0%; they tie on "Reach Dual"; Kimi is near the frontier pack on "Reach Multi-Modal"; and "Drawer close" is bad across the board.
The takeaway: aggregate scores hide structural per-task differences, and only the task-level breakdown reveals which labs are deliberately training for embodied/robotics scenarios.
Related event: Kimi K3.1 Pro Matches Top Tier on Robot Benchmarks at 20x Cost(3 posts)→
More from Embodied
- Robot fights hit the streets in China as humanoid brawls go viral — iamfakhrealam · 2026-10-26
- Gemini 4 Argon Turns One Video Into a MuJoCo Sim, Apollo 2 Hits 7 Straight Shots — TechPreacher · 2026-10-26
- Alibaba's λ0 humanoid learns from GoPro videos, hits 62.5% on Unitree G1 vs π0.5's 40% — AIFlow_ML · 2026-10-26
- Open-source sky camera run by Claude Code catches 17 meteors overnight, cuts star false alarms to 7% — Terrible_Award_2124 · 2026-10-26
- Apple asks suppliers to boost orders 30% as foldable iPhone shipments raised to 8-10M units — asymco · 2026-10-26
- Indian workers are paid to film haircuts and cooking as robotics training data — Olivier__OG · 2026-10-26