Robobench evals: Astra reasons with just 100 tokens vs 400-1k for rivals; Qwen 3.8 27B and GLM 5.3 Flash lead open models
arankomatsuzaki · x · 2026-09-28
Researcher arankomatsuzaki shares open-vs-closed model comparisons on Robobench and adjacent robotics tasks, measuring supervisor/executive capability. The standout finding beside the score gap: Astra consumes only 100 tokens for reasoning, while other models burn 400-1k even at low reasoning levels. Among open models, Qwen 3.8 27B and GLM 5.3 Flash performed best, close to Grok and Muse, while hy-embodied-vlm-1.0 and embodied r1.5 lagged far behind. He sees a narrow window to study the capability ramp of pre-physically-intelligent LLMs before the gap shrinks.
Related event: Researchers say now is the window for open models to catch up in robotics(2 posts)→
More from Embodied
- Developer's New Robotics BLDC Controller PCB Batch Arrives, Running Unit Tests — wavefnx · 2026-11-16
- YC-backed Ornadyne's flapping-wing drone was classified as a bird by Air Force detectors — import_jmr · 2026-11-16
- Uber starts offering Amazon's Zoox robotaxi rides in Las Vegas — Polymarket · 2026-11-16
- Ant Lingbo Embodied AI Challenge crowns TARS team after 48-hour real-robot hackathon — 新智元 · 2026-11-16
- Tesla FSD fleet tops 7 billion city miles, adding ~500M in under four weeks — XFreeze · 2026-11-16
- Ropedia Releases Xperience 2: 1,500+ Scanned Real-World Scenes Paired With Egocentric Demos — liuziwei7 · 2026-11-16