Training your own LLM judge for ~$5 in 1h40 on one GPU, 89.5% accuracy
MaziyarPanahi · x · 2026-11-15
Maziyar Panahi shares a complete low-cost recipe for training an LLM judge: $5, 1h40 on a single GPU, reaching 89.5% accuracy on 4,035 RAG and safety test rows. Key steps: 40K labeled decisions capped at 2,505 per task; dropping 5,645 rows overlapping the test set to avoid leakage; Qwen3.5-4B with Unsloth decision head and LoRA on just 1.4% of weights; calibration on 400 held-out rows (accuracy 88.75%, ECE 0.034). Final eval: RAG 89.16%, safety 89.78%. A directly reproducible cheap alternative to LLM-as-judge pipelines.
More from coding & agent
- 159k Lovable Apps Scanned: Many Expose Chats and Passport Data via Misconfigured Supabase APIs — bibryam · 2026-11-15
- Rust + wgpu rewrite renders 250k-file dependency graph at 60fps on M3 — IgorCarron · 2026-11-15
- Agents Orchestration Is Mostly Token Furnaces Now, Argues Dev — pvncher · 2026-11-15
- DataTalksClub Founder Launches Full-Time AI Academy in Germany, Tuition State-Funded — Al_Grigor · 2026-11-15
- How far can a company take AI? Greg Isenberg's 49-point playbook — FinanceYF5 · 2026-11-15
- TensorFold PR fixes MCP servers blocked by macOS sandbox over dyld framework loading — AIFlow_ML · 2026-11-15