Grok Build tops HF Harness Leaderboard in all four configs but fails on text-only models
TheZachMueller · x · 2026-11-18
Joel Niklaus added xAI's coding agent Grok Build to the Hugging Face Harness Leaderboard: same models, same 198 coding tasks (Terminal-Bench 2.1 and DeepSWE), three attempts each, only the harness varies. On GLM-5.3-Flash and DeepSeek-V4.1-Flash at low and high reasoning effort it places 3rd, 1st, 2nd and 1st — the only harness of 11 to make the top 3 in all four, though statistically tied with 6-7 others each time. On text-only Ripple-65B-Preview a habit costs it: its file tool tells the model "You see image files as images," so the model believes it is multimodal, sends an image, the server refuses, and Grok Build ends the attempt on the spot — happening in 38 cases.
More from coding & agent
- Agora launches AI Developer Toolkit: Skills and CLI let coding agents build RTC apps — testingcatalog · 2026-11-19
- Perplexity launches Context API: clean markdown from URLs at $1 per 1,000 fetches — perplexity_ai · 2026-11-19
- 28,200 LLM Judgments Show Why You Shouldn't Let Claude Code Design Your Judges — dl_weekly · 2026-11-19
- Writing Code Is Cheap Now; Issue-to-PR Is the Bottleneck — Enter the Software Factory — Pavan_Belagatti · 2026-11-19
- AI Puts Rewrites Back on the Menu — If You Get Good at Verification — rseroter · 2026-11-19
- Replacing a $150K/yr media buyer's weekly ad review with 10 Claude skills across 4 ad platforms — aryanXmahajan · 2026-11-19