Cross-provider LLM-as-Judge eval pipeline built in n8n, open sourced

ZtasnimS75 · reddit · 2026-11-19

The author built an LLM-as-Judge evaluation pipeline in n8n using two independent judges from different providers to eliminate self-family preference, with judge disagreement tracked as its own reliability metric.

Design

Limitations: 5 questions is a demo with no statistical significance; human labels are starter values; a judge failing to return valid JSON scores 1, which can understate the candidate. Workflow JSON and SQL are open sourced; the author asks whether to add a third judge or a tie-break rule.

Original post →

More from coding & agent

coding & agent channel →