Grok 4.5 tops VulcanBench v3 with 91.3% on real repo engineering tasks
XFreeze · x · 2026-07-23
Grok 4.5 tops VulcanBench v3 for repository-level software engineering
- A screenshot of BenchLM’s public scoreboard shows Grok 4.5 at 91.3%, ahead of GPT-5.6 Sol at 87.0% and Claude Fable 5 at 87.0%.
- VulcanBench v3 is described as a demanding benchmark for real-world repository-level engineering tasks, not simple code snippets.
- The tasks are based on real merged pull requests across Python, Rust, TypeScript, JavaScript, and Go.
- Solutions are checked with deterministic hidden tests in isolated environments.
- The post says Grok 4.5 solved 21 of 23 tasks, claiming it can handle multi-file changes, debugging, and harder engineering work that needs more than one-shot answers.
Related event: Grok 4.5 Tops VulcanBench Software Engineering Leaderboard(2 posts)→
More from coding & agent
- Spring AI 2.1.0-RC1 adds tool-call confirmation, pausing agent loops before real-data access — therealdanvega · 2026-10-26
- 250 Builders Hack With Codex at Berlin Community Hackathon Ahead of OpenAI DevDay — paw_lean · 2026-10-26
- Building AI bots on raw APIs? Devs say models are 'dumb without the harness' — madeofAjala · 2026-10-26
- HarnessMM cuts agent tokens per task by up to 88% while raising success rates — omarsar0 · 2026-10-26
- Microsoft demos intent-classifying agents built with Copilot App and Foundry, deployable across Teams and Outlook — marlene_zw · 2026-10-26
- Looking for a wearable mic with best battery life that feeds recordings to personal agents via MCP — altryne · 2026-10-26