LAGEN: one frame of latency can cut real-time agent performance by 95%

wzenus · x · 2026-10-22

A research thread introducing LAGEN (Latency-Aware Agents):

An invisible 1-frame delay leaves agents with as little as 4.2% of zero-delay reward on Flappy Bird; 2 frames drop fast ball-catching success from 64% to 0%. OpenVLA, π0.5 and GR00T all collapse — faster inference alone won't fix real-time control.
Method: training can't reproduce random network spikes (large batches slow episodes), so LAGEN replays target inference delays during training via "latency profiles", comparing Fixed/Normal/IID/Temporal models; the Temporal profile predicts real-GPU performance within 3.81% error, 3x better than assuming constant delay.
Latency-aware training improves all 12 real-time game and embodied tasks; high-latency training transfers to low latency but not vice versa.
Surprises: simply writing the delay into the prompt ("Latency: 150 ms") lifts Flappy Bird multi-latency return from <20 to 434; GPT-6.1 Sol playing Asterix by watching the screen scores 5.8k zero-shot vs 3.6k for a trained OpenVLA at the same latency.

More from Embodied