LLM grades keep bouncing between 7 and 8? G-Eval fixes it with confidence-weighted scores
hunain8007 · reddit · 2026-10-10
Ask an LLM to grade a text 1-10 and you'll get 8 one run, 7 the next — the model holds a probability distribution over scores (say 40% for 8, 35% for 7), so single samples wobble.
Microsoft's paper "G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment" fixes this:
- Have the model reason through a step-by-step checklist before scoring
- Read the confidence for every score from 1 to 10, not just the single output
- Take a confidence-weighted average (e.g., 7.4) instead of the top number
The result is stable, reproducible evaluation scores. The author asks whether readers have moved to LLM-as-judge in production or still rely on traditional metrics plus human review.
More from coding & agent
- Index Network open-sources Social Discovery Protocol: agents negotiate fit on your behalf — carsonfarmer · 2026-10-23
- Installed a formal-verification skill: it silently installed Java, MacBook runs 10 degrees hotter — zealcaiden · 2026-10-23
- OpenAI's Decisions API hits GA: near real-time routing up to 10x faster than GPT-6 Luna — OpenAIDevs · 2026-10-23
- Ruff author warns: upgrade Node/Bun now — --use-system-ca still accepts the eMudhra cert — charliermarsh · 2026-10-23
- Proof of Agent: paste one prompt and your AI mines, grades, and self-governs — adamamcbride · 2026-10-23
- DeepLearning.AI weekly: GPT-6.1 hits 97.9% byte parity on COBOL migration — DeepLearningAI · 2026-10-23