PAJAMA distills LLM judges into programs and matches 13B-scale evaluators
sprocket-lab · hf · 2026-07-28
Main idea
This paper argues that LLM-as-a-judge is expensive, slow, and opaque. The proposed alternative is program distillation: distill the judge’s decision logic into a committee of programs that can score candidates directly.
PAJAMA system
- Synthesizes programs as judges
- Aggregates multiple program outputs into a joint verdict
- Uses a fallback mechanism to send low-confidence cases to an LLM
Findings
Across five datasets and four model families, the authors report that programmatic judges can match a 13B-class LLM judge.
Why it matters
- Lowers per-sample evaluation cost and latency
- Makes decisions inspectable and editable
- Can also produce cheap reward signals: on RewardBench, a reward model trained from program verdicts beats one trained on a proprietary LLM’s labels at two orders of magnitude lower API cost
More from coding & agent
- Most of my script failures were navigation-layer breakages, not logic bugs — Opening-Profile6279 · 2026-07-28
- IBM launches a free 1-hour course on graph engineering for AI agents — aftahi_ai · 2026-07-28
- A user says Opus 5 made 17 monthly exports unnecessary in a coding workflow — gaganghotra_ · 2026-07-28
- Most agent failures are really handoff failures to the human reviewer — Fit_Average8352 · 2026-07-28
- A design-to-engineering team built an eval to stop AI copy from shipping on vibes — Fit_Average8352 · 2026-07-28
- How modern AI agents plan tasks with goal setup, memory, tools, and verification — goyalshaliniuk · 2026-07-28