Specialized model beats GPT-6 Astra at captioning egocentric robot videos, best frontier model only 43.5% correct
kimberlywtan · x · 2026-10-13
A partnership between tkkong's team and MeckaAI tested whether frontier models can label egocentric videos for robot learning: five frontier models captioned 200 clips, graded by humans. The best performer, GPT-6 Astra (max reasoning), got only 43.5% of clips fully correct. The chart shows the specialized Mecka model hitting 79.5%, far ahead of GPT-6 Astra (43.5%), GPT-6.1 Sol (38.0%), Claude Opus 5.5 (28.5%), Claude Fable 5.1 (26.5%), and Gemini 3.8 Flash (24.5%). The authors argue dense video captioning remains a task where specialized models outperform general-purpose foundation models, with many open problems left in robotics.
Related event: Frontier Models Struggle to Label Robot Egocentric Videos(2 posts)→
More from Embodied
- WUJI's robot hand learns pen spinning entirely in simulation, demos glove teleoperation — chris_j_paxton · 2026-10-13
- Ultra Robotics raises $50M Series A; one human shepherds 6 robots handling 4,000-6,000 shipments a day — chris_j_paxton · 2026-10-13
- Robotaxi extends hours to 6am-11pm, adds Grok in Cybercab — yunta_tsai · 2026-10-13
- Sim-to-real skepticism: pixel-space native training beats generated simulators, argues DeWitte — chris_j_paxton · 2026-10-13
- Collov Labs launches NewEyes AI, a personal agent with eyes — Scobleizer · 2026-10-13
- Amazon now has over 1 million robot workers. What's next for human employees? — lulzxdxdxd · 2026-10-13