Sub-Agent Benchmarking on 8GB VRAM
soteko · reddit · 2026-07-17
The author runs a **sub-agent benchmark for reading codebases** in his PHP AI platform, aiming to find an auxiliary model for the main model (GLM 5.2) that can run on **8GB VRAM** and is specialized for file search and editing. ### Evaluation Setup System prompt requires the sub-agent to use only read-only tools (`read_file`, `list_dir`, `search_files`) to explore the codebase and answer key file paths, method signatures, and the full flow from enqueue to handler in the Queue subsystem. ### Results Summary - Multiple models were compared on the same task, including `gemma-4-12b-it`, `gemma-4-e4b-it`, `qwen3.5-9b`, `ornith-1.0-35b`, `qwen3.6-35b-a3b@q5_k_xl`, `gemma-4-26b-a4b-it-qat` - In the first two rounds with `--steps=6`, **ornith 1.0 35b**, **gemma 4 26b**, and **qwen3.6 35b** scored highly, while `qwen3.5-9b` directly failed - The third round with `--steps=9` is ongoing for higher step comparisons (screenshot shows partial results) ### Key Takeaway This post is not about model size but about **how to build a reproducible benchmark for codebase sub-agents and select the best local model for retrieval/file editing**.
More from coding & agent
- Why vector databases slow AI agents down after constant writes — PrajwalTomar_ · 2026-07-21
- A 13-minute GitHub Copilot video digs into prompt caching — lee_stott · 2026-07-21
- Codex turns out 123 screensavers in one playful batch — intellectronica · 2026-07-21
- Autoresearch proposes packaging ML runs as studies with questions, analysis, and code diffs — morgymcg · 2026-07-21
- CHAP defines approvals, handoffs, and audit logs for human-agent workflows — DeliveryTechnical199 · 2026-07-21
- The author says Codex reached 20x and is now debugging spec decoding on a hybrid parallel setup — TheZachMueller · 2026-07-21