GPT-6 Astra Drives a Real Car via openpilot, Claude and Grok Fail

DrivingBench released full results of a test in which four general LLMs drove a rented Toyota Corolla through comma's openpilot ($999) via an MCP cloud interface, completing an 130m cone course. GPT-6 Astra finished in 5:22 on its second attempt while Claude and Grok failed entirely. Each model got 3 attempts within one continuous chat, relying on in-context learning. If confirmed, general models may have acquired real driving ability without specialized training.

Confirmed

• DrivingBench published complete results: real car control via openpilot, 130m course, 3 attempts per model in one continuous conversation.
• GPT-6 Astra completed the cone course in 5:22 on its second attempt; Claude and Grok failed.
• Astra appears to have completed the task without driving-specific training.

Unconfirmed

• Ben Todd speculates Astra's pretraining data may already include driving and robotics data, possibly with RL, though likely entirely in virtual environments; this attribution remains speculative.
• A Substack comment relayed by Ben Todd questions the "stepwise improvement" narrative: scoring counts distance traveled over 3 attempts, but course difficulty is not progressive, with only a few choke points.
• Ben Todd's claim that the capability lacks Waymo/Tesla-style hand-engineered rules is his personal judgment.

Why it matters

• If general models truly emergently drive without specialized training, it reinforces the scaling-over-specialization thesis — paulnovosad quipped "the bitter lesson gets more bitter" — with direct implications for autonomous driving roadmaps.
• The benchmark is among the first to hook general LLMs to a real vehicle in the physical world; Claude and Grok's collective failure shows the capability remains limited to top-tier models.
• Methodological critiques remind the field that emergent-capability claims need stricter scoring design and difficulty control.

2026-09-23 ~ 2026-09-25 · 6 related posts

• [source] Astra appears to drive a car with no specialist training, DrivingBench publishes full results — ben_j_todd · 2026-09-23
• Ben Todd: Google Astra Suddenly Gained Car-Driving Ability Without Specialist Training — ben_j_todd · 2026-09-23
• Driving ability emerged suddenly in frontier models, likely from virtual RL — ben_j_todd · 2026-09-24
• [source] Benchmark Score Jumps Misleading: Critique of Cone-Course Model Evaluation — ben_j_todd · 2026-09-24
• Bitter Lesson Strikes Again: Astra Drives Cars With Zero Domain Training — paulnovosad · 2026-09-24
• [source] GPT-6 Astra becomes first LLM to pass a real driving test, beats Claude and Grok in rented Corolla — 新智元 · 2026-09-25