Grok 4.5 tops VulcanBench v3 with 91.3% on real repo engineering tasks
XFreeze · x · 2026-07-23
Grok 4.5 tops VulcanBench v3 for repository-level software engineering
- A screenshot of BenchLM’s public scoreboard shows Grok 4.5 at 91.3%, ahead of GPT-5.6 Sol at 87.0% and Claude Fable 5 at 87.0%.
- VulcanBench v3 is described as a demanding benchmark for real-world repository-level engineering tasks, not simple code snippets.
- The tasks are based on real merged pull requests across Python, Rust, TypeScript, JavaScript, and Go.
- Solutions are checked with deterministic hidden tests in isolated environments.
- The post says Grok 4.5 solved 21 of 23 tasks, claiming it can handle multi-file changes, debugging, and harder engineering work that needs more than one-shot answers.
Related event: Grok 4.5 Tops VulcanBench Software Engineering Leaderboard(2 posts)→
More from coding & agent
- Stanford's PaperFormalizer turns research papers into Lean-verified versions with author in the loop — StanfordUniversity · 2026-10-27
- Hugging Face TRL makes score centering + masked importance sampling default in async GRPO — _lewtun · 2026-10-27
- AutoResearch agent's weird training curve gets dug into instead of discarded — ZimingLiu11 · 2026-10-27
- Firecrawl offers 2.5x credits to a16z Speedrun founders, unveils $75M Series B — devdigest · 2026-10-27
- Chums: open-source Apple Watch app puts Claude Code and Codex on your wrist — brandon_galang · 2026-10-27
- Open-source Clairic Studio turns pasted text into typed data and working code in 5 languages — dfinke · 2026-10-27