Open 4B Model Beats Jev 1.14 Judge in 22-Minute, $1.87 Training Run
MaziyarPanahi · x · 2026-10-21
- The author trained his own evaluation judge on the open Jev datasets, using the 130K-row Jev-RAG-Claim-Verification dataset on Hugging Face (RAG claim verification / hallucination detection samples with gold labels, predictions, token usage and cost fields).
- Results: on 4,035 held-out RAG + safety calls, the 4B open model scored 89.4%, beating Jev 1.14's 82.2%.
- Cost: 22 minutes on a single NVIDIA H200 via Hugging Face Jobs for $1.87; running Jev 1.14 through its API for all 6,035 calls cost just 20 cents.
- Twist: on PubMed answers the model never saw, Jev wins — hinting at possible overfitting on Jev's own data. Both datasets are fully open.
Related event: Token Arcade Turns LLM Structured Output Into a Playable Game(3 posts)→
More from Research
- Going Meta dives into ontologies and graph semantics for production knowledge graphs — JeremyCMorgan · 2026-10-22
- Cohere's Rachad Alao on how open science underpins transparent, secure foundation models — PyTorch · 2026-10-22
- Four Gemini agents beat Minecraft's Ender Dragon in 27 hours — dying 605 times — krishnan · 2026-10-22
- Ropedia W0: a world model learning loco-manipulation from 10k hours of human egocentric video — _akhaliq · 2026-10-22
- Stanford colloquium on AI in oncology features senior author of Nature paper on autonomous medical AI agents — jonc101x · 2026-10-22
- UofT paper AREa: adaptive receptive-field attention for efficient high-res diffusion transformers — CSProfKGD · 2026-10-22