Snorkel AI's RIFT Framework Audits Rubric Quality Across Five Public Benchmarks

ajratner · x · 2026-11-18

Snorkel AI senior research scientist Ankit Aich will present RIFT (RubrIc Failure mode Taxonomy) on Dec 15. Rubrics grade model/agent outputs and supply reward signals in training but are rarely audited themselves. RIFT computes quality signals from rubric content and maps deviations to named failure modes; in controlled tests with defects injected into 60 expert-authored rubrics, both linear models and frontier LLMs detected the failures. Applied to 147 tasks across five public benchmarks (finance, medicine, software engineering, professional advice, reasoning), RIFT surfaced many missing/misaligned criteria plus large amounts of hackable and repeated rubric items.

Original post →

More from coding & agent

coding & agent channel →