Gemma 4 E2B's verbalized confidence collapses to 3 values; independent judge ranks passages better

clduab11 · reddit · 2026-09-22

The author ran a controlled experiment on a homebrewed RAG pipeline: Gemma 4 E2B (run locally via llama.cpp) writes answers, while a judge scores retrieved passages — comparing Gemma's own verbalized confidence vs. the independent Jev scorer.

Key findings

Gemma's verbalized confidence collapses onto three values (0.00/0.50/1.00 account for 99.6% of 3,885 judgments; only 6.3% fall between 0.10–0.90), leaving almost no threshold mobility (11% vs 99% for Jev).
On 473 paired passages, Jev ranks gold-answer-bearing passages far better: AUROC 0.926 vs 0.804 (+0.122). Of 189 passages Gemma marked 1.00, only 40% actually contain the gold string, yet Jev still separates within that subset (AUROC 0.835).
End to end, however, the Jev-judged pipeline scored 0.612 against a no-judge baseline of 0.740, missing its pre-registered bar — it lost mainly by declining far more questions.

Setup: 500 SimpleQA questions, self-hosted SearXNG search, claude-sonnet-5 grading (+1 right / 0 declined / -1 wrong). The author discloses that Jev is a paid API he built; the writing, search and memory stack is fully local, and the judge can be swapped for any local SFT classifier.

Untried local fixes: token logprobs on the verdict, or vote fraction over 10 samples. The stated claim is deliberately narrow: small-model verbalized confidence leaves a threshold nothing to move.

Original post →

Gemma 4 E2B's verbalized confidence collapses to 3 values; independent judge ranks passages better

More from coding & agent