Testing Qwen3.8-27B inside a coding agent: evaluation plan
Binary_orchid · reddit · 2026-08-25
The author shares an evaluation plan for Qwen3.8-27B running locally within the EvoX coding agent harness. The focus is on the model's behavior in complex tasks: repo reading, tool calling, error recovery, and multi-turn consistency, rather than ranking desktop apps.
Planned tasks include:
- Screenshot-to-page: Comparing Low/Medium/XHigh reasoning levels in a clean frontend repo, measuring first render time, correction turns, and total time.
- USGS Earthquake Dashboard (Advanced): Testing rendering, filtering, and interactions with static GeoJSON and live feeds.
The author controls variables (e.g., disabling experience reuse) and measures reasoning token counts and time-to-first-tool-call to balance quality and latency. The community is invited to critique the plan.
More from coding & agent
- Are the fastest GPU kernels now written by AI? Kernels are hyper-verifiable, argues one dev — fleetwood___ · 2026-10-16
- Anthropic lead engineer's 4-question prompt makes agents more ambitious, now a free skill — daniel_mac8 · 2026-10-16
- ChainForge dev shows 'feature map' for orchestrating subagents on large PRs — IanArawjo · 2026-10-16
- aiDotEngineer benchmarks agent orchestration pipelines, staging humans as Foreman and workers — vikvang1 · 2026-10-16
- Developer mass-produces 200 videos in hours using Claude Code + Hyperframes — mcraddock · 2026-10-16
- Penny University hosts software factory discussion; dev shares $40/mo vendor-agnostic AI stack — JnBrymn · 2026-10-16