Testing Qwen3.8-27B inside a coding agent: evaluation plan
Binary_orchid · reddit · 2026-08-25
The author shares an evaluation plan for Qwen3.8-27B running locally within the EvoX coding agent harness. The focus is on the model's behavior in complex tasks: repo reading, tool calling, error recovery, and multi-turn consistency, rather than ranking desktop apps.
Planned tasks include:
- Screenshot-to-page: Comparing Low/Medium/XHigh reasoning levels in a clean frontend repo, measuring first render time, correction turns, and total time.
- USGS Earthquake Dashboard (Advanced): Testing rendering, filtering, and interactions with static GeoJSON and live feeds.
The author controls variables (e.g., disabling experience reuse) and measures reasoning token counts and time-to-first-tool-call to balance quality and latency. The community is invited to critique the plan.
More from coding & agent
- Vercel Goes Full Backend: Python GA, Containers, and Daemons — cramforce · 2026-10-16
- Datadog Debuts Scout 1.0, a Domain Model for Incident Investigation — tianyin_xu · 2026-10-16
- WorldMemArena Accepted at NeurIPS: Memory Storage ≠ Memory Use — xwang_lk · 2026-10-16
- Vercel Ship Demos Container Deploys for Go, Ruby, Java and More — edwin · 2026-10-16
- TRL adds day-0 support for fine-tuning Muse Spark Open's 816B MoE with LoRA on 64 H100s — QGallouedec · 2026-10-16
- Databricks engineers land impactful PRs on day 2 with AI dev tools, vs week 3 before — Yuchenj_UW · 2026-10-16