AutoRubric, a Rubric-Science-Backed LLM Evaluation Tool, Heads to COLM 2026
deliprao · x · 2026-09-21
AutoRubric—combining rubric science best practices with LLM-as-a-Judge research for structured evaluation of non-verifiable domains—has been accepted to COLM 2026. It lets you define weighted criteria (including negative scores for wrong claims), validate against human labels, and iterate via built-in meta-evaluation; an open-source Python library, cookbook, and example grading NMC vs LFP battery answers with gpt-5.1-mini are available.
More from coding & agent
- SemiAnalysis: Persistent agents now eat 45% of our AI spend, and Perplexity's harness beats Claude Tags — dylan522p · 2026-11-12
- Ora's Ranker 3.0 grades whether agents can actually use your site; scores drop 40+ points — yenkel · 2026-11-12
- Gemini API Managed Agents now support self-hosted sandboxes, incl. Vercel — steren · 2026-11-12
- Rayline launches ARC, a per-turn model router claiming 90% cost cuts — testingcatalog · 2026-11-12
- Karpathy's llm-wiki: a pattern for building personal knowledge bases with LLM agents — stochasticchasm · 2026-11-12
- Agent self-improvement with frozen weights: failure traces add just 4pp, blind search gets 60% — burny_tech · 2026-11-12