Robotics eval breakdown: Kimi K3.1 lands between frontier models but costs 20x more
teortaxesTex · x · 2026-10-26
teortaxesTex breaks down a per-task cost-vs-success chart of a robotics agent eval (9 tasks, uniform 40-step budgets): K3.1 sits roughly between Gemini Sol 6 and GPT-1017. 1017 is the only model at 0% on 'Drawer open'; on 4 of 9 tasks both 1017 and K3.1 score 0%; they tie on Reach Dual; Kimi is near the frontier pack on Reach Multi-Modal (86-89%); but it drops to 56% on 'Drawer close'.
The core issue isn't capability but price: Kimi is bigger and had high-quality vision earlier, so it should do well on robotic/spatial/agent tasks — yet it costs 20x more. Sol 6.1, the author argues, must now be the north star for Chinese models.
Related event: Kimi K3.1 Pro Matches Top Tier on Robot Benchmarks at 20x Cost(3 posts)→
More from Embodied
- Indian workers are paid to film haircuts and cooking as robotics training data — Olivier__OG · 2026-10-26
- Robot dog walks a real dog in San Francisco; owner hints at secret humanoid project — beffjezos · 2026-10-26
- AI short 'Daybreak' generated on a single NVIDIA DGX Spark with Imagination Studio — Scobleizer · 2026-10-26
- NVIDIA shows RTX Spark powering sim-to-real robotics: Microduck robot learns somersault in simulation — Aiden_Tech_Ai · 2026-10-26
- Motubrain 2.0 livestreams half a month of full Physical AI training pipeline — jiqizhixin · 2026-10-26
- Open-source 3D-printed litter-picking robot MOSS tackles beach sand, gets stuck on big dunes — Scobleizer · 2026-10-26