Realtime-Bench: agents score just 44% when the world moves while they think
jiqizhixin · x · 2026-11-22
A team from UT Austin, NVIDIA, Ohio State, UIUC and Northwestern introduces Realtime-Bench (arXiv), evaluating agents in dynamic environments where a correct action can fail if the world changes while the model is thinking. Best Wall Clock success rate is 44%, best Pass-cubed (all 3 trials) is 36%; more reasoning has mixed effects.
The benchmark compares three ways of advancing time:
- Wall Clock: includes actual inference latency (serving-dependent);
- Paused Clock: zero delay, idealized reference;
- Token Clock (proposed): advances simulated time by generated token counts (Δt = n/κ), isolating timing decisions from physical serving speed.
Tasks are evolving Game/Web UIs (fixed-FPS screenshots + Set-of-Marks) where the agent outputs GUI actions between frames.
More from Research
- 10 months after a Reddit question on AI solving Millennium Prize Problems, claims abound — haider1 · 2026-11-22
- Fable 5.2 rewrites model weights so an E2E driving bot learns left-side driving — 4310sy · 2026-11-22
- IceCube citizen science challenge draws 4,000+ classifications to help train AI — CatAstro_Piyush · 2026-11-22
- Michael Levin teases Morphoceuticals, his bioelectricity regeneration startup — drmichaellevin · 2026-11-22
- Reddit manifesto argues cybernetics shows humanity never escaped nature — Cyborgized · 2026-11-22
- Neshter: a fully local tool to visualize, measure and prune attention heads in small LLMs — Bayqus · 2026-11-22