Blind test of 5 local AI agents on one Qwen model: scores range from 3/5 to 5/5
GapNew4766 · reddit · 2026-10-25
A Redditor ran a blind test of five local AI agents on an M5 Max MacBook Pro (64GB), all driving the same Qwen 3.8 27B (Q4KM) via llama.cpp across five identical everyday tasks: merging messy contact lists (46 people, two Alex Kims), renaming a function across three files, a bug fix with hidden tests, counting 43 errors in log files, and converting a PDF receipt to a spreadsheet, with results verified by script.
Results
- A: 5/5 in 10 min, fastest, but installed Python libraries without asking
- B: 5/5 in 24 min, slowest (15 min on the receipt alone), also installed without asking
- C: 3/5 — wrong format on the bug fix, failed the receipt because it asked permission to install and nobody answered
- D: 5/5 in 12 min, solid and well-explained, but oddly asked what the user wanted to call it mid-task
- E: 5/5 in 11 min, clean throughout
Takeaway: contacts, rename, and error-count tasks were solved by all five; the bug fix and receipt split them. A 5/5 agent that takes twice as long or modifies your system unasked isn't automatically best — and a 3/5 caused by over-caution isn't automatically worst. Candidates: Atomic Agent, goose, Hermes, OpenClaw, OpenCode; the answer key comes in a day or two.
More from coding & agent
- Slop Eisley: a Star Wars fan game fully coded by Claude Opus and Codex opens beta with public invite codes — jasonkneen · 2026-10-25
- Weekend opsec: hand Claude a checklist to audit your home network and devices — zedlander · 2026-10-25
- Cornell paper: 64.7% of AI agent rollouts melt down on errors, no adversary needed — 2C_ornot2C · 2026-10-25
- WildAWM synthesizes open-source computer-use agent worlds; Qwen3.6 gains +10.6 across 8 benchmarks — HuaxiuYaoML · 2026-10-25
- Meme: Claude as Sisyphus endlessly pushing the 'Some Rust Port' boulder — burny_tech · 2026-10-25
- Running Claude Code as PM over Codex workers to wire xAI Grok in as a coding agent — umeshonai · 2026-10-25