HERMES pairs every file with its own AI model, hitting 94.2% on SWE-bench with a local 8B editor

alex_verem · x · 2026-10-12

Researchers at the University of Illinois and Penn State built HERMES, a coding system that pairs each file in a codebase with its own resident AI model that knows only that file, messaging dependent files when it changes something. Two other jobs go to separate models: one selects which files a task involves, another reads test failures to localize the culprit.

In their experiments, file-level editing ran on Qwen3-8B via Ollama on their own GPUs, while GPT-5.6 Sol handled planning and diagnosis. That mix resolved 94.2% of SWE-bench Verified (GPT-5.6 Sol doing everything scored 97.0%), with the gap under 4.5 points across four benchmarks and 26.2% lower cost on Terminal-Bench 4.0. Running everything on Qwen3-8B still resolves 80.6% of 500 real GitHub issues at $0.13 per task.

The takeaway: the harness matters more than expected — GPT-5.6 Sol scored 6.5% on whole-repo migration inside Codex but 31.0% inside HERMES, an average +12.4 points over standard agent setups. The tradeoff: HERMES costs more per task on the same model ($1.80 vs $1.15), but the expensive model only plans and diagnoses while a self-hosted small model does the editing.

Original post →

HERMES pairs every file with its own AI model, hitting 94.2% on SWE-bench with a local 8B editor

More from coding & agent