FutureSim Q3: LLMs now genuinely improve from experience, Opus 5.5 tops at 50%
rickasaurus · x · 2026-11-16
The FutureSim benchmark (replaying real-world events to test forecasting past models' knowledge cutoff) released Q3 2026 results showing a discontinuous change: last-gen LLMs genuinely get better from experience.
- Q1 2026: even the best models barely improved after feedback; top accuracy 25-27%, most open-weight models had negative Brier skill scores (worse than abstaining).
- Q2 2026: modest progress; GPT 5.6 Sol led at 31% accuracy.
- Q3 2026: across-the-board jump — Opus 5.5 tops with 50% accuracy and 0.35 Brier skill score, shredding the Q2 SoTA; GPT 6.1 Sol hit 47%, DeepSeek V4.1 Flash 1017 reached 41%. The author notes Opus 5.5 called 50+ subagents and wrote 400+ research notes over the 90-day simulation while being 50% more token-efficient than 6.1 Sol.
Built by a team from ELLIS Tübingen/Max Planck Institute et al. (arXiv:2605.15188), the results suggest test-time adaptation is now improving industry-wide — a key signal for measuring long-horizon, open-ended agent adaptation.
Related event: Opus 5.5 Tops FutureSim Q3 Benchmark with 50+ Sub-Agents(2 posts)→
More from Models
- Stealth Anthropic model spotted in Devin build as METR chart shows frontier pace up 1.4x — daniel_mac8 · 2026-11-16
- Eval Firm Trajectory Told 40 Agents "NEVER Stop" in Harness Without Telling Epoch — peterwildeford · 2026-11-16
- METR time-horizon chart updated: doubling every 1.7 months since Opus 4.5, 'Model 4' ~38h — ben_j_todd · 2026-11-16
- Gemini Pro user says its coding agent rejects every Three.js/SVG prompt as 'illegal' — Necessary-Buy-7410 · 2026-11-16
- 28-model tracking table: half publish no training cutoff, Gemini 4 Carbon 8 months stale at launch — Joozio · 2026-11-16
- Opus 5.5 caught using account email to identify user in supposedly memory-less session — birchlse · 2026-11-16