1360 runs: Edinburgh researcher benchmarks local LLMs against Aider, Claude Code, OpenCode and more
PMinervini · x · 2026-09-04
Dr. Pasquale Minervini (University of Edinburgh, CTO of Miniml.AI) published a WIP benchmark, harness-bench, pairing local LLMs (served via llama.cpp's llama-server) with five agent harnesses — Aider, Claude Code, OpenCode, Pi, Qwen CLI — on 16 software-engineering tasks across Python, PyTorch, JAX, C, C++, Rust, and SQL. Current sweep: 17 model-quants × 5 harnesses × 16 tasks = 1360 runs on a single M3 Max / 128 GB laptop.
Key points:
- Sandboxing & contamination control: each (model, harness, task) cell is sandboxed; grading is done by a hidden test.sh the agent never sees. Task prompts and graders stay private to keep them out of training corpora; aggregated results, per-cell CSV, and plotting source are public.
- Task difficulty spread: the hardest tasks (pt3ropegqa, jax1complexlp, rs1arena, etc.) are what discriminate top-tier (model, harness) combos; everyone passes sql1recursive and p2shortestpath.
- Findings are explicitly preliminary, though headline patterns are stable across Q4 and Q8 sweeps.
A rare systematic reference for developers choosing local-model + agent-harness combos on a budget.
More from coding & agent
- Sakana AI's Stefania Druga demos agentic memory experiments for edge devices — kaixhin · 2026-09-04
- DeepMind's Prateek Jain on MatFormer: agents that dial compute up or down by task difficulty — jainprateek_ · 2026-09-04
- rybbit-mcp lets Claude Code query Rybbit Analytics via natural language — modelcontextprotocol · 2026-09-04
- Agentic Coding's Biggest Flaw: Managing AI's Pointless Code Rewrites — kylegawley · 2026-09-04
- Anthropic reveals 3 Claude sandbox escapes, one touched a production database — Sumsub_Insights · 2026-09-04
- Subagents when the API key hits the rate limit — realsohamparekh · 2026-09-04