LLM-as-judge stuck on one confidence value? Switch to labels.
pedrinho_k2 · reddit · 2026-08-28
A developer found that LLM-as-judge evaluations were getting stuck on a single confidence score (0.72), rendering it meaningless. Anchoring scores in the prompt failed. The fix was counterintuitive: asking for categorical labels (coinflip/slight/clear/decisive) instead of numbers and mapping them in code. The same logic applied to checking if answers converged: ask for a boolean in code, not a comparison in text. Anecdotal data suggests the judge is most decisive on creative tasks, preferring precise, quantified language over evocative prose.
More from Research
- Developer Publishes Handbook on RAG and Context Engineering Based on Real Papers — techNmak · 2026-08-28
- Schmidhuber: 1st backprop-trained CNN for vision from 1988 — SchmidhuberAI · 2026-08-28
- Qwen3-8B fine-tuned with LoRA+GRPO mimics a famous ML blogger, fooling every AI detector — OtherRaisin3426 · 2026-08-28
- Fieldwork data is messy: AI challenge of unstructured reconstruction — anthara_ai · 2026-08-28
- Eval anti-cheat idea: serve models a stale HF cache from before grader fixes — willcb · 2026-08-28
- Study of 400k Claude Code sessions: Domain expertise beats coding skills — alex_verem · 2026-08-28