$1.87 fine-tune of Qwen3.5-4B beats Jev 1.14 as LLM judge, 89.4% vs 82.2%
MaziyarPanahi · x · 2026-10-21
- Developer MaziyarPanahi took 10K rows from each Jev dataset, fine-tuned Qwen3.5-4B with Unsloth for 1 epoch on a single NVIDIA H200 via Hugging Face Jobs—22 minutes, $1.87.
- On 4,035 held-out RAG + safety calls, his open 4B judge scored 89.4% vs Jev 1.14's 82.2%, with all models evaluated on the same 6,035 held-out rows against human labels.
- Twist: on unseen PubMed answers Jev wins—unless the small model learns from Jev. Model is open on Hugging Face.
Related event: Token Arcade Turns LLM Structured Output Into a Playable Game(3 posts)→
More from Models
- Baseten releases open-source decision models elephant-1 and mammoth-1, free for a week — baseten · 2026-10-22
- Recursive SI open-sources Flint, a local coding model hitting 125+ tokens/sec on MacBook Pro — CodeByPoonam · 2026-10-22
- Recursive's Flint: a 40B local coding model built in 4 months, 125+ tok/s on a MacBook — jeffclune · 2026-10-22
- NVIDIA Nemotron demos go live — mervenoyann · 2026-10-22
- vLLM adds day-0 support for Flint: 500 TPS on a single GB200 — _rockt · 2026-10-22
- WIRED: Recursive's Flint shows AI-built AI is getting closer to real recursive self-improvement — cong_ml · 2026-10-22