Local models + a code-side gate: 54% → 92% accuracy and zero silent wrong answers on table tasks
Shoddy_Total9784 · reddit · 2026-11-16
An informal A/B test on r/LocalLLaMA ran three local Qwen2.5 models (1.5B/3B/7B GGUF via llama.cpp on CPU only) on simple table questions (sum, average, min/max, filters).
- Arm A: the model writes Python, which is executed directly.
- Arm B: the model emits a tiny command DSL; code parses it, a gate validates the command against a catalog, and a deterministic runner computes the result — out-of-catalog requests are rejected with a reason. The idea: "code decides, the model explains."
Gate v8 results (same tasks, seeds, temperature; gate code hash-locked):
- Overall accuracy: A 54.0% vs B 92.4% (366/396)
- Fresh holdout: A 49.1% vs B 83.3%
- Silent wrong answers: A 51 vs B 0 (16 vs 0 on holdout)
- Tokens −68%, API-equivalent cost −74%
Honest limitations: single domain, not peer-reviewed, tasks and holdout written by the same team, TCO depends on assumed watts/electricity, and an earlier word-list gate (v7) lost its lead on rephrased questions (45.4% vs 47.2%) — which is why v8 validates the model's command instead of the request's wording.
Raw records, hashes, and the full MIT-licensed report are public in the HF dataset acik-teknik/skillpack-tests-2026-11-16; the authors ask for criticism on methodology.
More from coding & agent
- Dev Says AI Coding Tools Make Him Stop Caring About Folder Structure — prasenx · 2026-11-16
- Blogger builds Elon Musk interview archive site in 1 hour with Tencent Hy4 flash, open-sources two Skills — vista8 · 2026-11-16
- VibePrevisBench: 242-task benchmark tests coding agents on 4D Blender scene generation — Kling-Keye · 2026-11-16
- raylib author threatens to block users over AI-generated GitHub issues — yacineMTB · 2026-11-16
- Dev open-sources local Apple Health MCP server that feeds workouts to AI agents and pushes plans back to Apple Watch — PrinzPwnage · 2026-11-16
- Coding agents crack a 1994 Jeep's ECU via a $3 Arduino and start controlling it — LeviTurk · 2026-11-16