Untrained Qwen beats purpose-trained decision model by 35 points in real repo test
KingPinX · reddit · 2026-09-23
Several "typed decision model" projects popped up on reddit with benchmark claims, so the author scored them on his own private git history (a Discord RPG bot), using conventional commit prefixes as free labels — stripping the type(scope): prefix so models don't get an answer key.
Results (with a free baseline each task):
| Task | Jev (hosted) | SemIf (CPU) | Von 1.0 | free baseline |
|---|---|---|---|---|
| commit type, 6-way (n=90) | 78.9% | 62.2% | 26.7% | 31.1% regex |
| file routing, 12-way (n=80) | 81.3% | 63.8% | 8.8% | 48.8% keyword |
| is-a-feature (AUC) | 0.944 | 0.899 | 0.513 | 0.543 keyword |
Key findings:
- SemIf (MIT) trains nothing: it loads a stock Qwen3.5-4B, stuffs state and option descriptions into one prompt, and reads option logits off a single forward pass. This zero-training approach beat purpose-built Von by 35 points on commit type and 55 on routing — Von's 8.8% routing lost to keyword overlap (48.8%) where random gets 8.3%.
- Von dissected: it labeled 69 of 90 commits refactor; feed it a period or ten random digits and it still says refactor. Its feature-probability sign is even flipped.
- Confidence gates do not transfer: SemIf's 0.85 gate lifted routing 22 points but commit type only 4.5; the same gate on Von kept 30% of rows and got 0% right — it selects for confidently wrong. Gates must be tuned per task and per model.
- Cost: 30.9s per decision on CPU for the longest task (p95 41s); latency tracked state length, not option count. SemIf hardcodes ngpulayers=0, so all numbers are CPU.
The repo is private, but the method transfers: conventional-commit labels plus an afternoon of work. With n=80–180, gaps under 10 points are noise. Next up: testing Laya and others.
More from coding & agent
- Half of PM job listings now ask for evals skills; Ramp, Shopify, Harvey, Cursor share gains — HamelHusain · 2026-09-23
- Agent tasks can spiral into surprise bills: user calls for spend caps and auto-stop — yi111 · 2026-09-23
- Security audit: autonomous research program XBOW credited with ~12 upstream bug fixes — moyix · 2026-09-23
- Telling agents to use formal verification helps them write better code — sh_reya · 2026-09-23
- Team uses Typesafe's Jev to prompt follow-up questions during ticket creation — TheMoonMidas · 2026-09-23
- The most common evals mistake: skipping error discovery and measuring the wrong thing — FinanceYF5 · 2026-09-23