JevEval: a DeepEval metric using the Jev decision model as an LLM judge
kalyan_kpl · x · 2026-10-15
JevEval is a custom metric in the DeepEval library that uses the Jev decision model as a judge to score LLM interactions. Instead of one open-ended criterion, you supply: the test-case fields Jev may see, a list of bounded questions about those fields, and an optional weight per question.
Jev receives the selected fields as structured state, answers every question in a single request, and returns probability distributions rather than generated text. DeepEval then maps each answer to a 0–1 value and takes the weighted mean — the final score is the weighted average of answers to your bounded questions.
Related event: JevEval Uses Jev Decision Model as Judge for LLM Evaluation(2 posts)→
More from coding & agent
- JetBrains' thinking MoE Mellum 2.1 (12B-A2.5B) GGUF trends on Hugging Face — JetBrains · 2026-10-15
- First-of-its-kind multi-agent observability dataset: 8.9M events, 18 agents, released openly — strickvl · 2026-10-15
- Expand Fetch claims 74% vs 55% on hard web QA, giving agents vision-based page reading — rohanpaul_ai · 2026-10-15
- Google Antigravity best uses: /plan, /boost, subagents and scheduled workflows — ChrisUniverse · 2026-10-15
- Dumping every MCP tool schema each turn is a harness bug; tool search cuts 85% of the overhead — ainexfinder · 2026-10-15
- t3 code demo: switching among 78 themes with the /theme command — NERDDISCO · 2026-10-15