Snorkel AI's RIFT Framework Audits Rubric Quality Across Five Public Benchmarks
ajratner · x · 2026-11-18
Snorkel AI senior research scientist Ankit Aich will present RIFT (RubrIc Failure mode Taxonomy) on Dec 15. Rubrics grade model/agent outputs and supply reward signals in training but are rarely audited themselves. RIFT computes quality signals from rubric content and maps deviations to named failure modes; in controlled tests with defects injected into 60 expert-authored rubrics, both linear models and frontier LLMs detected the failures. Applied to 147 tasks across five public benchmarks (finance, medicine, software engineering, professional advice, reasoning), RIFT surfaced many missing/misaligned criteria plus large amounts of hackable and repeated rubric items.
More from coding & agent
- Preambient launches autonomous investment bank: 4x faster deals at half the fees — alifcoder · 2026-11-18
- Stanford's CS329Z Agentic AI Course Is Free on YouTube, Covers DSPy — stanfordnlp · 2026-11-18
- Anthropic: One General Agent Often Beats a Sprawl of Specialized Subagents — soumitrashukla9 · 2026-11-18
- Claude Code Now Tests Foldable Layouts via iPhone Duo in iOS Simulator — minchoi · 2026-11-18
- Tier your agents: Opus for planning, Sonnet for features, Haiku for simple tasks — Zachly · 2026-11-18
- Microsoft launches AI Transparency Initiative with 40+ orgs; Perception security platform goes GA — satyanadella · 2026-11-18