Jev fails as an LLM scorer on OntBench: rates almost everything positively, contradicting human and Codex ratings
amplifiedamp · x · 2026-09-18
A developer tried using Jev as an automated scorer in OntBench (a benchmark for how well different LLMs build ontology maps) and found it unusable: it scores almost everything very positively, disagreeing with both the author's manual ratings and Codex's ratings.
The full five-level scoring prompt (from 'multiple major mistakes' to 'supported claims with explicit limits') is shared, asking the community what went wrong. It's a textbook LLM-as-judge failure: the scorer lacks discrimination and skews positive.
More from Models
- 'You Can Feel the Gradient Slopes': Dev Contrasts Benchmaxxed RL Models vs General Ones — theteknosaur · 2026-11-02
- LightOnOCR-3 Launches: Tops FRBench, Adds Layout Structure and Chart Extraction — IgorCarron · 2026-11-02
- NVIDIA's Kumo Is a Family of Four Tabular Foundation Models, Not Just One — pandeyparul · 2026-11-02
- SemiAnalysis: safety classifiers are underrated — Meta's Muse tops cyber benchmark at 35.2% — ThomasScialom · 2026-11-02
- DeepSeek V4.1 Flash 1017 wins over a longtime closed-source-only user: fast, cheap, reliable — _bachrc · 2026-11-02
- Local quants vs Claude on 9 real private bugs: 3-bit costs thinking time, not score — FeydRowan · 2026-11-02