Using an LLM as benchmark scorer fails: over-optimistic ratings diverge from human judgment
amplifiedamp · x · 2026-09-18
The author tried using Jev as a scorer on OntBench, a benchmark for how well different LLMs build ontology maps — and it just doesn't work. It scores almost everything very positively, disagreeing with both the author's manual ratings and Codex's blinded ratings.
The author shared their scoring prompt asking whether they misused it, noting that Jev now tracks Codex fairly closely overall but remains overly optimistic about everything.
More from Models
- 'You Can Feel the Gradient Slopes': Dev Contrasts Benchmaxxed RL Models vs General Ones — theteknosaur · 2026-11-02
- LightOnOCR-3 Launches: Tops FRBench, Adds Layout Structure and Chart Extraction — IgorCarron · 2026-11-02
- NVIDIA's Kumo Is a Family of Four Tabular Foundation Models, Not Just One — pandeyparul · 2026-11-02
- SemiAnalysis: safety classifiers are underrated — Meta's Muse tops cyber benchmark at 35.2% — ThomasScialom · 2026-11-02
- DeepSeek V4.1 Flash 1017 wins over a longtime closed-source-only user: fast, cheap, reliable — _bachrc · 2026-11-02
- Local quants vs Claude on 9 real private bugs: 3-bit costs thinking time, not score — FeydRowan · 2026-11-02