Custom benchmark: 35B Qwen3.6 scores 95% vs 53% for 120B GPT-OSS on coding agent
pauliusztin · reddit · 2026-09-26
A developer building an AI agent from scratch found that Qwen3.6-35B hit 95% pass@1 on his custom coding-agent benchmark, while GPT-OSS-120B — roughly 4x larger — managed only 53%, showing generic leaderboards and model specs can't be trusted for your own use case.
Benchmark design
- 19 tasks (7 easy/6 medium/6 hard) styled after Terminal-Bench, each with an instruction, a seeded repo, and a hidden verifier giving 0/1 reward
- The agent works in a sandbox and returns work as a git branch; hidden tests are injected only after the run to avoid leaking hints
- Verifiers are code, not LLM judges: modifying the wrong file or changing more than 8 lines in a file fails the task
- Opik handles observability, dataset versioning, and experiment tracking, making model comparison as simple as diffing two experiments
Key caveat: the harness was tuned specifically for Qwen3.6-35B, which explains the gap — optimizing your harness for your model matters at least as much as specs.
Cost finding: on the same workload, pay-per-token (OpenRouter, $0.18 per 1.2M tokens) beat pay-per-GPU-hour (Modal); GPU billing only makes sense if you can batch tasks and keep utilization near 100%.
Bottom line: model size and leaderboard rankings are only useful up to a point — test agents in your own use cases before trusting generic reports.
More from coding & agent
- Sydney Codex Community Buildathon set for Nov 14 with Cerebras-backed ultrafast compute track — paw_lean · 2026-11-11
- shadcn's improve hits 10k stars: let top models plan, cheap models execute code fixes — shadcn · 2026-11-11
- OpenAI open-sources Codex Security CLI and SDK for AI-driven vulnerability scanning and fixing — openai · 2026-11-11
- Claude Opus 5.5 runs an architecture search and finds AttnRes-Canon, a new top Transformer design — SonglinYang4 · 2026-11-11
- 5 research habits worth knowing: read papers backwards, reproduce first, wire coding agents via MCP — cneuralnetwork · 2026-11-11
- 'Adam can now do CAE': AI agent showcase in engineering simulation draws buzz — burhop · 2026-11-11