Astra hits ≥98% on 10 of 37 benchmarks, near-saturating the time-horizon eval suite
dswg97 · x · 2026-09-09
The author describes their time-horizon (TH) estimation methodology: injecting synthetic long-horizon (>2 hour) tasks at 0% success rate, then filtering bootstrap samples with non-trivial success-rate variance for a more reasonable estimate.
Key caveat: Astra is close to saturating the suite — scoring ≥98% on 10 of 37 benchmarks, versus GPT-5.5 saturating only 4/37 — making a confident TH estimate for Astra hard to produce.
Related event: Astra nears benchmark saturation, scoring 98%+ on 10 of 37(2 posts)→
More from Models
- Kimi K3.1 Pro tops MathArena open models, closing frontier gap to about 6 months — teortaxesTex · 2026-10-26
- Ruff creator Charlie Marsh: frontier models find infinite bugs in any software — charliermarsh · 2026-10-26
- Dev testing one skill across Opus 5.5 and Sol 6.1 finds instruction following 'all over the map' — lucasmeijer · 2026-10-26
- supermemory unveils learner-1, claiming continual learning for any model — julianweisser · 2026-10-26
- Anthropic says Opus 5.5 cuts cost per task by ~40% vs Opus 5 — taufiqintech · 2026-10-26
- AI search APIs ranked: top 12 within 7 points, yet prices differ nearly 7x — FinanceYF5 · 2026-10-26