A $1.87-trained open 4B judge beats Jev 1.14 on 4,035 held-out calls
MaziyarPanahi · x · 2026-10-21
Developer Maziyar Panahi trained his own LLM judge on the Jev datasets: Qwen3.5-4B + Unsloth, 22 minutes on a single H200 via Hugging Face Jobs, costing $1.87. On 4,035 held-out RAG + safety calls it scored 89.4% (RAG 87.9%, safety 90.7%) vs Jev 1.14's 82.2%; untrained Qwen3.5-4B scored 71.4%, and a version distilled from Jev's own answers hit 83.3%. The twist: on 2,000 unseen PubMed judgments, Jev (916) beats the human-label-trained model (855) — only the Jev-taught 4B (922) stays ahead, suggesting generalization depends on the teacher's signal.
Related event: Token Arcade Turns LLM Structured Output Into a Playable Game(3 posts)→
More from Research
- Paper asks 'How long is the coast of a neuron?' — resolution shapes neural computation — neurovium · 2026-10-21
- OpenAI scales thousands of AI agents to find new leads in biological data — truecakesnake · 2026-10-21
- EAU: Co-evolving user simulator, environment, and RL setup for training LLM agents — UAlberta · 2026-10-21
- Meta's Self-Credit: Dense Rewards from Sparse RLVR via Intra-group Rescaling — meta · 2026-10-21
- New paper: game-theoretic limits of being helpful yet harmless when an AI assistant doesn't know who it's talking to — conitzer · 2026-10-21
- CS1: a 4B clinical decision model trained on 12.4M questions beats open rivals — AI4Pathology · 2026-10-21