Realtime-Bench: agents score just 44% when the world moves while they think

jiqizhixin · x · 2026-11-22

A team from UT Austin, NVIDIA, Ohio State, UIUC and Northwestern introduces Realtime-Bench (arXiv), evaluating agents in dynamic environments where a correct action can fail if the world changes while the model is thinking. Best Wall Clock success rate is 44%, best Pass-cubed (all 3 trials) is 36%; more reasoning has mixed effects.

The benchmark compares three ways of advancing time:

Tasks are evolving Game/Web UIs (fixed-FPS screenshots + Set-of-Marks) where the agent outputs GUI actions between frames.

Original post →

More from Research

Research channel →