Using JEV as judge: LLM citations never fully check out, ~8 sources per task
pizzababa21 · reddit · 2026-09-22
The author used JEV as a judge to check whether LLM-cited sources actually support the claims in web search retrieval tasks, with a strength score. Models typically cite around 8 sources per task, and no test run had all 8 pass. With plain search and no rules, JEV even caught outdated information by noticing article dates months old despite the user asking for recent updates. The most common failure it caught: statements drawing on multiple sources while citing only one. The author is optimistic this will make next-gen models considerably more reliable via better-vetted AI-generated test data.
More from Models
- antirez backs low-bit DeepSeek 4.1 Flash over Qwen 27B dense on Apple M5 Macs — antirez · 2026-11-12
- Gemini re-renders images as text when vision is removed, dropping only 6% WSR — zacharynado · 2026-11-12
- Bare-harness coding benchmark: Fable 5.2 best overall, Gemini 4 Carbon best value — Semantic_meaning · 2026-11-12
- Google's Physical-Engineering's Last Exam: all 14 frontier models score below 50% — dl_weekly · 2026-11-12
- Reddit's 2026 LLM hall of fame: Opus 5.5 still unbeaten after 9 rival releases — G0dZylla · 2026-11-12
- Commit leak reveals unreleased Gemini 4 variants: Argon, Hydrogen and Helix (unconfirmed) — BlackHC · 2026-11-11