n8n workflow evaluates LLMs with two cross-provider judges to avoid self-preference
ZtasnimS75 · reddit · 2026-11-19
A developer built and open-sourced an n8n workflow that evaluates an LLM using independent LLM judges:
- A 5-question test set with reference answers and human scores;
- Candidate models (gpt-oss-20b and gpt-oss-120b via Groq) answer each question;
- Two judges from different providers (Gemini Flash Lite and Nemotron Lightning, free via OpenRouter) score answers 1-5 against references;
- Code computes pass rate, judge-human agreement, mean absolute error, and judge disagreement;
- Results go to Slack, email and Postgres.
Key design: neither judge is the candidate model, avoiding self-preference; a judge gap of 2+ points surfaces as an explicit metric. The author notes 5 questions is only a demo — real evals need a bigger labeled set. Workflow JSON, setup and SQL are on GitHub.
Related event: Open-source n8n pipeline uses cross-vendor LLM judges(2 posts)→
More from coding & agent
- Building agents is easy, scaling them for thousands of customers is where the real work hides — Franc0Fernand0 · 2026-11-19
- Deepnote rebuilds data apps: build with Codex or Claude, host in Deepnote — alex_verem · 2026-11-19
- NDepend open-sources a C# MCP Server with a complete tutorial and 35-minute video — adnan_hashmi · 2026-11-19
- Creator Studio ships with in-app assistants, MCP support, and open Keynote format — jasonwuishere · 2026-11-19
- Vercel's agent-desktop lets Claude Code drive native Mac apps via the accessibility tree — Arindam_1729 · 2026-11-19
- Anthropic publishes free 33-page guide on turning repetitive prompts into reusable Claude Skills — aftahi_ai · 2026-11-19