Open-source rubrify compiler mirrors typesafeai's Jev abstraction, dev says autorubric predates it by 8 months
secemp9 · x · 2026-09-21
Developer secemp9 found that typesafeai's Jev uses an abstraction strikingly similar to autorubric, released 8 months ago and to be presented at COLM next month — Jev's Noul/Score/Choice criterion types map 1:1 to autorubric's types. He has now open-sourced rubrify, an independent reimplementation from first principles.
rubrify is a rubric compiler and judge engine for LLM evaluation:
- Define structured evaluation rubrics as typed Python objects, compiled into immutable bundles
- Runs criterion-by-criterion LLM-based judgments against text responses
- Supports evolving rubrics via GEPA's reflective prompt optimization on human-annotated datasets
- Built on harnai for multi-provider LLM access (OpenAI, Anthropic, DeepSeek, Google, local proxies); install via pip install rubrify, evolution needs rubrify[evolve]
More from coding & agent
- Grok Bot team shipped 11,000 PRs in a month: reliable agents come from checks, not prompts — sophiamyang · 2026-11-12
- Ora's Ranker 3.0 grades whether agents can actually use your site; scores drop 40+ points — yenkel · 2026-11-12
- Netlify ships Server: a plain-language map of serverless functions vs long-lived servers vs VMs — thisiskp_ · 2026-11-12
- Gemini API Managed Agents now support self-hosted sandboxes, incl. Vercel — steren · 2026-11-12
- Rayline launches ARC, a per-turn model router claiming 90% cost cuts — testingcatalog · 2026-11-12
- Karpathy's llm-wiki: a pattern for building personal knowledge bases with LLM agents — stochasticchasm · 2026-11-12