Realtime-Bench: Even the Best Model Scores Just 36% on Real-Time Agent Decisions
ManlingLi_ · x · 2026-10-31
- New benchmark Realtime-Bench evaluates agent decision-making under real-time constraints.
- Best model (GPT-6-Astra) scores only 36%; the fastest model (Jev, answering in under 1s) scores just 8%.
- Key finding: more reasoning doesn't always mean better agents — extra thinking helps only when the clock is paused; with a real clock, slow thinking costs the win.
- Even with time frozen, many tasks remain unsolved.
- The team raises a core question for agent evaluation: optimize reasoning quality alone, or decision quality under time pressure? Benchmark is playable online.
More from Models
- Anthropic kills the 5-hour limit and renames Max 20x to Ultra; users suspect a nerf is coming — doodlestein · 2026-10-31
- Cartesia's cascaded voice agents top the Speech Agent Arena, beating native speech-to-speech — rohanpaul_ai · 2026-10-31
- Correction: newer model gets worse on the extreme-hard tail, pass@5 crossover predicted — teortaxesTex · 2026-10-31
- Hyperflow H3 hits #5 on I2V and #3 on video editing at under half Seedance 2.5's price — ArtificialAnlys · 2026-10-31
- Dev publishes FLARE-based Qwen 3.8 Flash Next images with 1M context for RTX 6000 PRO — johannes_bertens · 2026-10-31
- Rumored GPT-6.5 'Bel' sparks proposal for a no-rules game test over rehearsed launch demos — VraserX · 2026-10-31