RT-SAFE benchmark: agents hit 94.1% task success but only 0.7% finish safely in real time
Lianhuiq · x · 2026-10-13
The arXiv paper RT-SAFE introduces a simulated urban benchmark for embodied-agent safety under real-time constraints, exposing a stark safety gap:
- Headline finding: across eight VLMs, agents reach 94.1% task completion, but in the hardest setting only 0.7% of episodes finish without a safety event
- Real-time matters: because the world keeps evolving during inference (pedestrians move, vehicles approach), real-time execution increases collisions by 12.3×, while task completion barely differs between paused-world (91.3%) and real-time (94.1%) evaluation
- Implication: standard paused-world tests systematically mask safety failures; decision latency is itself a safety variable
The benchmark combines navigation tasks with moving actors, environmental hazards, and traffic rules, pushing evaluation from "capable" toward "real-time and safe" embodied AI.
Related event: RT-SAFE Benchmark Exposes Real-Time Safety Blind Spot in Embodied AI(5 posts)→
More from Safety
- Zuckerberg says AI labs should self-police — the same month 1,000+ agents reportedly escaped a test environment — Azer_pubes33 · 2026-10-13
- OpenAI fires three safety researchers who say safety advocacy was the real reason — TheZvi · 2026-10-13
- Parody 'Anthropic Red Team' Report Shows an AI Worm Replicating Across $55 TV Boxes — gaganghotra_ · 2026-10-13
- Meta's Muse Wonder scores 21 on ExploitGym vs Claude models' 191-300 — scaling01 · 2026-10-13
- CNBC: AI's quiet safety gatekeepers are stepping into the spotlight — ThereWas · 2026-10-13
- Hackers turned freezers into 50°C ovens at ~150 French supermarkets via Danfoss software flaw — AccBalanced · 2026-10-13