SpecJudge v0.2.0: Forcing LLMs to Cite Real Evidence Fixes Total 8B Model Breakage
jokiruiz · reddit · 2026-08-06
Open-source CLI tool SpecJudge released v0.2.0, designed to recommend the most cost-effective AI models based on project specs. The core update forces the LLM judge to cite actual text fragments supporting its ratings, preventing the model from using fluent narratives to mask incorrect assessments.
While building regression tests, the developer discovered a critical bug: 8B parameter local models failed across all test projects. The root cause was that Ollama's format: json guarantees valid JSON but not the specific JSON structure requested, leading models to output [true] instead of a citation ID. By enforcing a strict JSON Schema, the success rate jumped from 0/9 to 9/9. Additionally, the new version pins sampling parameters to ensure reproducible evaluation results.
More from coding & agent
- Matt Pocock Shares /wait-what Coding Agent Skill for Plain English Explanations — mattpocockuk · 2026-08-06
- Developer Shares Claude Skills Tip: Using /wait-what to Quickly Explain Code — mattpocockuk · 2026-08-06
- Opinion: AI Agents Desperately Need a Global Orchestrator and Centralized Approval Page — koltregaskes · 2026-08-06
- Procedurally Generating Joseon Architecture with Claude and Three.js — nptacek · 2026-08-06
- Cloudflare OS Architecture: Lying to AI Agents to Ensure Execution Safety — jedisct1 · 2026-08-06
- DeepSeek API Surprises Developers with Built-in Web Search Integration — max_paperclips · 2026-08-06