SpecJudge author argues LLM-as-judge tools should never ask models for self-reported confidence

jokiruiz · reddit · 2026-11-04

The author of open-source SpecJudge uses a local LLM as judge to recommend models for a project: every rating must cite a spec fragment or it's discarded, and the model is deliberately never asked how confident it is — self-reported confidence from a small model isn't evidence about the project.

The problem: the current three states (sufficient / scarce / refusal) are too coarse — a project missing its constitution and one resting on a single cited sentence both land in scarce.

Hard rule: no 0-100 score. Percentages invite comparing incomparable projects and lend unearned precision to a decorated guess, which then gets trusted more than a blunt one.

Two candidate designs:

A) A defined ratio like 4 of 6 with each signal marked ✓/✗ — inspectable but secretly equal-weighted: "nothing grounded" and "one source trimmed" both cost one point
B) An ordinal scale (solid / qualified / thin / nominal) with components exposed in JSON so users can re-weight

The six signals: every dimension grounded in a citation; no unsupported dimensions; citations spread across the project; hardest requirement is a stated constraint; real work artifacts exist; context budget admitted every source.

The author leans toward A but keeps the equal-weighting objection open, and asks readers which format they'd act on versus misread.

Original post →

SpecJudge author argues LLM-as-judge tools should never ask models for self-reported confidence

More from coding & agent