New egocentric benchmarks show SOTA VLMs fail ~1/3 of robot next-action predictions
mattbeane · x · 2026-10-20
Animesh Garg's team released two new egocentric benchmarks evaluating VLM readiness for robotics, with the headline finding that semantic reasoning ≠ perceptual timing.
MicroPlanBench (consequential next-action planning): predicting the single valid next atomic action from 8s of egocentric video still fails 1/3 of the time — GPT-6 Astra hits 65.6%, GPT-6.1 Sol 65.5% at 1/5th the cost. GPT-6 Astra error breakdown: 60% wrong manipulation, 23% wrong hand assignment, 17% wrong object/destination; 42% of failures involve tracking the wrong hand.
MicroAnnotationBench (continuous temporal grounding): probes estimating time to a predicted state. Quoting the post, a transformer model doing next-frame prediction plus time-to-state estimation was seen as impossible six months ago — though the quoter notes this still isn't robotics' bottleneck.
Related event: MicroPlanBench: No SOTA VLM Beats 70% on Robot Next-Step Planning(3 posts)→
More from Embodied
- Embodied First Exam benchmarks coding agents on 1,090 tasks across 44 robots — ManlingLi_ · 2026-10-23
- Delivery robot spotted cruising city sidewalks on its own — HomeNowWTF · 2026-10-23
- Dev builds ESP32 desk display showing Claude Code and Codex agent status via local Go helper — tonylook · 2026-10-23
- Dream Labs emerges from stealth with Dream-1 robot foundation model at $3/hour — scott_e_reed · 2026-10-23
- AdaDex turns seconds of human hand motion into generalizable dexterous robot skills — mangahomanga · 2026-10-23
- Perplexity's Portable Computer lands on AMD Ryzen AI Max, running local 27B model — AravSrinivas · 2026-10-23