qwen-code proposal: 30-50 feature-type SWE tasks to benchmark a coding agent's own harness features
yiliang114 · ghdev · 2026-10-13
Issue #14011 on QwenLM/qwen-code proposes a "harness SWE" benchmark suite: 30-50 original software-engineering tasks in Terminal-Bench/Harbor format, designed specifically to elicit qwen code's own feature surface — todo tracking, plan mode, skills/workflows, subagent delegation, background tasks, memory, and permission gates.
Key design points:
- Task shape maps to feature: multi-step dependent implementations trigger todo tracking; ambiguous cross-module refactors trigger plan mode; mechanical changes over 10 files trigger subagent delegation; long builds trigger background tasks; destructive/.env-touching tasks trigger permission gates; cross-session preferences trigger memory.
- Verifiers follow the DeepSWE principle: test observable behavior only (paginated endpoint, passing tests, updated README), never implementation details, so any correct solution passes.
- The same tasks double as user-facing demos, and the repo's existing integration-tests/terminal-bench/ adapter can be reused.
More from coding & agent
- Isolating Antigravity chats per subfolder with AGY Rules and PreInvocation Hooks — bibryam · 2026-10-16
- Gemini shown off modding Minecraft to turn players into any Spider-Man — zacharynado · 2026-10-16
- Workflow: annotate UI screenshots in tldraw, hand to Claude, get better UI — max__drake · 2026-10-16
- Beyond chat wrappers: designing stateful, multimodal AI apps as interaction conductors — dr_cintas · 2026-10-16
- IBM engineer maps 5 levels of AI coding, from manual to a 3-day agent software factory — TejasKumar_ · 2026-10-16
- 10 open-source repos that let your AI agent reverse-engineer apps and games — victor_explore · 2026-10-16