Rubricon: scaling test-time compute to auto-generate expert-grade LLM rubrics
deliprao · x · 2026-11-07
A new paper introduces Rubricon, a rubric-generation pipeline that grounds rubric refinement in how candidate rubrics actually score example responses, rather than relying on LLM self-critique alone. Motivated by the open question of whether today's LLMs can write rubrics that reliably evaluate tomorrow's smarter models, Rubricon was benchmarked against 4 synthetic rubric methods across 5 benchmarks including ProfBench, ACE, PRBench and HealthBench, and its rubrics consistently align best with human expert scoring (Spearman ρ up to 0.5 on PRBench).
More from Models
- Deep Learning Weekly #480 ships with 'Gemini 4 Carbon' mention and model-routing paper — dl_weekly · 2026-11-07
- PostTrainBench: AI agents' post-training score jumps from 23% to 45% in 7 months — karinanguyen · 2026-11-07
- scaling01: The cope for Chinese models is just distillation from Fable — scaling01 · 2026-11-07
- DeepSeek leads the accuracy-cost Pareto frontier "not even close", says casper_hansen_ — casper_hansen_ · 2026-11-07
- ChatGPT leak: SSE stream exposes per-chunk retrieval scores, quotes worth 100 points — lilyraynyc · 2026-11-07
- $50K live trading battles: AI bots currently edge out Claude and GPT — Mr_Hodlerr · 2026-11-07