Local models + a code-side gate: 54% → 92% accuracy and zero silent wrong answers on table tasks

Shoddy_Total9784 · reddit · 2026-11-16

An informal A/B test on r/LocalLLaMA ran three local Qwen2.5 models (1.5B/3B/7B GGUF via llama.cpp on CPU only) on simple table questions (sum, average, min/max, filters).

Gate v8 results (same tasks, seeds, temperature; gate code hash-locked):

Honest limitations: single domain, not peer-reviewed, tasks and holdout written by the same team, TCO depends on assumed watts/electricity, and an earlier word-list gate (v7) lost its lead on rephrased questions (45.4% vs 47.2%) — which is why v8 validates the model's command instead of the request's wording.

Raw records, hashes, and the full MIT-licensed report are public in the HF dataset acik-teknik/skillpack-tests-2026-11-16; the authors ask for criticism on methodology.

Original post →

More from coding & agent

coding & agent channel →