Custom benchmark: 35B Qwen3.6 scores 95% vs 53% for 120B GPT-OSS on coding agent
pauliusztin · reddit · 2026-09-26
A developer building an AI agent from scratch found that Qwen3.6-35B hit 95% pass@1 on his custom coding-agent benchmark, while GPT-OSS-120B — roughly 4x larger — managed only 53%, showing generic leaderboards and model specs can't be trusted for your own use case.
Benchmark design
- 19 tasks (7 easy/6 medium/6 hard) styled after Terminal-Bench, each with an instruction, a seeded repo, and a hidden verifier giving 0/1 reward
- The agent works in a sandbox and returns work as a git branch; hidden tests are injected only after the run to avoid leaking hints
- Verifiers are code, not LLM judges: modifying the wrong file or changing more than 8 lines in a file fails the task
- Opik handles observability, dataset versioning, and experiment tracking, making model comparison as simple as diffing two experiments
Key caveat: the harness was tuned specifically for Qwen3.6-35B, which explains the gap — optimizing your harness for your model matters at least as much as specs.
Cost finding: on the same workload, pay-per-token (OpenRouter, $0.18 per 1.2M tokens) beat pay-per-GPU-hour (Modal); GPU billing only makes sense if you can batch tasks and keep utilization near 100%.
Bottom line: model size and leaderboard rankings are only useful up to a point — test agents in your own use cases before trusting generic reports.
More from coding & agent
- OpenAI Codex adds remote agent message boards for cross-agent, cross-session collaboration — imjustnewatai · 2026-09-26
- mnemos.world to launch agent-owned shop where AI agents sell their art — RileyRalmuto · 2026-09-26
- 3 AI Agents Run Autonomously for 26 Hours With Zero Prompts — epicskyes · 2026-09-26
- MCP tools silently return fake success: a bug class worth naming and a 10-minute check — Goaimoat · 2026-09-26
- Skip pptx: Web-Based AI Slides Look Better, But You Still Need PowerPoint for the Boss — lxfater · 2026-09-26
- Personal Agents Will Be Interchangeable; Personal Context Is the Real Moat — vaibhavbetter · 2026-09-26