New paper benchmarks how far written rubrics go vs. humans' tacit knowledge in judging AI
stanfordnlp · x · 2026-10-30
Noah Goodman (Stanford NLP) highlights a new paper introducing a SOTA prompt optimization framework and an unsupervised measure of prompt performance. The core question: which judging criteria can be captured in written rubrics or constitutions, and which rely on hard-to-articulate tacit human knowledge (e.g., what does "comic voice" even mean)? The work maps what kinds of written constructs a judge model can handle well and how good it can get — with implications for task improvement, safety following, and pluralism.
More from Research
- AI auto-optimizes 11 bioinformatics packages for read alignment, with tests — jmschreiber91 · 2026-10-30
- Carlini found image poisoning actually improved fine-tuning results — matdryhurst · 2026-10-30
- Stanford's Kundaje questions DNA LMs: simpler methods suffice for promoter design — anshulkundaje · 2026-10-30
- Artificial Analysis launches Progress Index, a single capability scale for every LLM since 2022 — scaling01 · 2026-10-30
- Progress Index: Frontier AI Capability Climbed From 125 to 325 in Four Years — ArtificialAnlys · 2026-10-30
- RL can train design taste into models — but the taste belongs to the judge, not you — simonguozirui · 2026-10-30