CMU paper: small model judges LLM outputs at 0.36% of the strongest judge's cost
akshay_pachaar · x · 2026-09-27
A new Carnegie Mellon paper tests whether small models like Jev can handle bounded evaluation decisions — grounding checks, instruction following, response preference — instead of full LLM-as-judge setups.
Key findings:
- Close on focused judgments: Jev stayed within 3 percentage points of the strongest judge tested on ordinary response preferences, evidence-grounded factuality, and final-answer checks.
- Massive cost gap: on the paper's matched workload, Jev cost only 0.36% of the strongest comparator's fee.
The practical takeaway: use cheap judges to evaluate far more agent responses across more criteria, escalating only the hard cases to stronger models.
More from Research
- Google Research open-sources PaperOrchestra, a multi-agent framework for writing AI papers — google-research · 2026-11-16
- Kind ICLR Reviewer Raises Score Despite Unresolved Concerns, Citing the Paper's Value — MarkSchmidtUBC · 2026-11-16
- Hand-editing 0.2% of weights plus 100 CPU-minutes turns an English TTS model into an 18MB Sanskrit one — kastnerkyle · 2026-11-16
- Why chirality matters for AI for Science: enantiomers behave differently in biology — madeofAjala · 2026-11-16
- RSS 2027 submissions open with FAQ clarifying two-stage review and negative-result bar — GeorgiaChal · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16