Benchmark Score Jumps Misleading: Critique of Cone-Course Model Evaluation
ben_j_todd · x · 2026-09-24
Ben Todd shares a Substack critique of a model's cone-course benchmark "step-like" score increase.
- Scoring gives the model 3 attempts and measures distance along the cone course; difficulty doesn't scale monotonically — progress hinges on a few chokepoints rather than gradual skill.
- Fable's third attempt nearly matches Astra's first; some cones are sparse/hard to see, and a sharp opening turn trips most models, including Fable's first two runs.
- Todd adds: the result isn't great yet, but at this rate of progress it could be within a year.
Related event: GPT-6 Astra Drives a Real Car via openpilot, Claude and Grok Fail(6 posts)→
More from Models
- Opus 5.5 feels nerfed — user blames system prompt restrictions, not the model — oyacaro · 2026-11-12
- Grok Bot now auto-detects and blocks prompt injection hidden in web pages — heypearlai · 2026-11-12
- ChatGPT now links to brand homepages far more, reshaping brand-mention tracking — lilyraynyc · 2026-11-12
- YC-backed Monte launches post-training platform to build proprietary models from your agents' real work — ycombinator · 2026-11-12
- Leak Claims OpenAI Will Launch Entire GPT-6.2 Family Tomorrow to Counter Opus 5.5 and Fable 5.2 — heypearlai · 2026-11-12
- ARC-AGI-3 leaderboard: models score 90%+ but at $10K+ per task — haider1 · 2026-11-12