Cross-provider LLM-as-Judge eval pipeline built in n8n, open sourced
ZtasnimS75 · reddit · 2026-11-19
The author built an LLM-as-Judge evaluation pipeline in n8n using two independent judges from different providers to eliminate self-family preference, with judge disagreement tracked as its own reliability metric.
Design
- Test set: 5 questions (facts/math/tech/reasoning) with reference answers and human scores
- Candidate: Groq's gpt-oss-20b and gpt-oss-120b, tested in separate runs
- Judge A: Gemini 3.5 Flash Lite; Judge B: free Nemotron 3.5 Lightning via OpenRouter; both score 1-5 and neither shares the candidate's family
- Metrics computed in code: judge average, absolute error vs human score, pass (avg ≥4), agreement with human (within 1 point), judge disagreement (≥2 points)
- Reports to Slack and email; items and runs stored in Postgres (Supabase)
Limitations: 5 questions is a demo with no statistical significance; human labels are starter values; a judge failing to return valid JSON scores 1, which can understate the candidate. Workflow JSON and SQL are open sourced; the author asks whether to add a third judge or a tie-break rule.
More from coding & agent
- Hermes adapter solves a SWE-Bench Pro task in 35 tool calls where the base Qwen3.8 model failed — yb2698 · 2026-11-19
- NousResearch's Hermes adapter lifts Qwen3.8-27B agent scores: SWE-Bench Pro 29.4%→43.1% at 4× fewer tokens — yb2698 · 2026-11-19
- One Amazon EBS gp3 command scales a 10M-vector Weaviate index 26×, from 9.8 to 260 QPS — philipvollet · 2026-11-19
- Human Principal launches private beta: cryptographic proof a human is behind your AI agent — yenkel · 2026-11-19
- GPT-6.2 builds code-only 3D fairy town and a deep-sea civilization in 10–20 minutes — OpenAIDevs · 2026-11-19
- Designers are being asked to do interior design while ignoring the architecture — round · 2026-11-19