Codifying the Judge: Scalable Evaluation via Program Distillation
Tzu-Heng Huang, Shengqi Qiu, Frederic Sala
cs.AI, cs.LG
2026-05-29
PAJAMA distills an LLM judge into Python programs, matching OLMo-2-13B accuracy at 47x the speed and beating GPT-4 labels as a reward signal at 50x lower cost.
LLM-as-a-judge is now the standard for automated evaluation, but it carries four familiar problems. The most direct is cost: running proprietary models like GPT-5 or Gemini-3-Pro over millions of samples runs up prohibitive API bills. It is opaque; the decision process is hidden, so you cannot tell whether a score follows its stated rationale or is a hallucination. LLM judges are systematically biased toward verbose answers, flashy formatting, and emotional language. And there is a re-inference tax: change the rubric and you rerun the LLM over the whole dataset.
What this paper does is distill that judging logic into executable programs, so at evaluation time you run code instead of calling an LLM.
The core is program distillation. PAJAMA uses a strong LLM (Claude Opus 4.6) to generate Python functions judgingfunction(query, response), each encoding one evaluation rubric as code and returning a numeric score. The authors curate 10 code-expressible rubrics (relevance, language quality, logical coherence, evidence density, and others), synthesize 80 candidate programs, and select the top-k by validation accuracy to form a committee (8 to 21 depending on the dataset).
A single program will not cover every input, so the committee is aggregated. Each program's output is min-max normalized, converted to a preference difference, and binarized into {-1, 0, +1} votes. A weak-supervision label model (Snorkel) then learns a per-program weight from the agreement patterns and emits a joint verdict.
For samples the committee cannot decide (high vote variance, or posterior near 0.5), a fallback escalates them to an LLM judge. The routing signal comes from the committee itself, with no extra model, and sweeping the threshold traces the full accuracy-throughput curve.
Across five preference datasets (JudgeLM-100K, PandaLM, MultiPref, Prometheus, Preference-700K), the program committee averages 78.11% accuracy, on par with OLMo-2-13B-Instruct (79.70%) and Qwen2.5-3B-Instruct, and about 8 points behind the strongest judge, GPT-5 Thinking (85.72%). On Prometheus, just 8 programs reach 88.78%, matching OLMo-2-7B.
The speed gap is larger. The committee runs 47.25x faster than OLMo-2-13B-Instruct, and 2.12x faster than even the smallest Gemma-3-270M-It while scoring 30.11 points higher; against larger 14B/12B models it is roughly 50x faster.
Using program outputs as routing signals to mix in LLMs pushes the accuracy-throughput Pareto frontier: paired with OLMo-2-7B, accuracy rises 5.0% at 2.9x throughput; with Qwen2.5-3B, up 2.6% at 2.2x.
The most practical use is as a reward signal. On RewardBench, a reward model distilled from program verdicts beats one trained on GPT-4 labels: 56.65 vs 54.98 on Prometheus training, 61.77 vs 57.28 on JudgeLM training, at about 50x lower API cost ($7.21 vs $363.97) and zero proprietary calls at evaluation time.
| Signal source | RewardBench avg | Labeling cost |
| GPT-4 labels (Prometheus) | 54.98 | $363.97 |
| PAJAMA programs (Prometheus) | 56.65 | $7.21 |
It replaces a per-sample, opaque LLM judge with transparent, inspectable, editable programs whose cost barely grows with dataset size. Fix a rubric and you edit a program, with no dataset rerun. For anyone running RLHF or large-scale benchmarks, that is real money saved. A byproduct is a cheaper signal than proprietary labels for training reward models.
It works only for rubrics you can encode as code (relevance, coherence, coverage). Fuzzy, subjective judgments that resist programmatic features still need the LLM. Synthesizing programs takes a strong model up front (Claude Opus 4.6), a one-time but not free step, and quality is bounded by it. Samples where the committee abstains get random labels for coverage, which injects noise. The authors note a small gap to an oracle router remains, and richer features could close it. And be clear: 78% is the 13B tier, not the GPT-5 tier; this is a cost-quality trade, not a wholesale quality win.