469-question eval of Qwen3.8-27B fine-tunes: best local model 28x slower than Opus 5.5 at 99.6%

norenEnmotalen · reddit · 2026-10-08

The author ran a 469-question domain-specific eval of Qwen3.8-27B fine-tunes vs frontier models under llama.cpp with uniform thinking settings:

Opus 5.5 topped the list at 99.6% accuracy; no model hit 100%. Best local pick: mradermacher/Signal-3.8-27B-Terse-Coder.i1-Q4KM — a Q4KM quant outperforming L/XL variants.
Speed gap is massive: 9 hours locally vs 19 minutes on frontier (28x slower), in exchange for 100% privacy vs 0 on API providers.
Astra shows the lowest token usage but its reasoning appears masked by the provider; author calls on OpenRouter to mandate quantization disclosure.
gpt-oss-120b badly failed pandas/numpy tasks — only domain-specific evals surface such blind spots.
Eval tool open-sourced at github.com/ashe-wb/tuieval.

Takeaway: run your own domain evals; the author proposes a "PTA index" (time/tokens/accuracy) as a personal benchmarking KPI.

More from coding & agent