TypeSafe's Jev accused of closely mirroring AutoRubric's evaluation abstractions from 8 months ago

On September 21, Delip Rao alleged that after investigating TypeSafe's (@typesafeai) new product Jev, he found its evaluation abstractions highly similar to his AutoRubric framework released about 8 months earlier and soon to be presented at COLM 2026: Jev's Noul/Score/Choice criterion types map almost one-to-one onto AutoRubric's criterion types. Ian Arawjo independently investigated and reached the same conclusion. As of now, TypeSafe has not publicly responded, and whether AutoRubric was directly borrowed remains unresolved.

Confirmed

• Both Delip Rao and Ian Arawjo noted the near one-to-one correspondence between Jev's criterion abstractions (Noul/Score/Choice) and AutoRubric's criterion types
• The AutoRubric paper was released about 8 months ago and accepted at COLM 2026; its repository and detailed technical report have been public since February. The work brings rubric-science best practices to LLM-as-a-Judge for structured evaluation of non-verifiable domains, both evaluating LLM outputs and using LLMs as evaluators
• Another user noted that a COLM paper's code closely resembles their independently built "rubrify"; Delip Rao acknowledged many data structures are similar but stressed his repository and technical report were public long beforehand

Unconfirmed

• No evidence yet on whether TypeSafe directly borrowed from AutoRubric; the Jev side has not responded publicly

Why it matters

• ShriramKMurthi offered a counterview: these criterion categories have long been standard statistical categories in social-science methodology texts, and Jev's real distinction lies in its architecture—suggesting "similar abstractions" may not amount to plagiarism
• The episode touches on attribution of research ideas in LLM-as-a-Judge being adopted by commercial products, unfolding right before the COLM 2026 presentation and potentially affecting community trust in evaluation tooling

2026-09-21 ~ 2026-09-21 · 7 related posts

• Researcher Says TypeSafe's Jev Mirrors His Autorubric LLM Eval Framework From 8 Months Ago — deliprao · 2026-09-21
• [source] Jev's eval abstraction maps 1:1 to autorubric paper from 8 months ago, researcher finds — deliprao · 2026-09-21
• [source] AutoRubric, a Rubric-Science-Backed LLM Evaluation Tool, Heads to COLM 2026 — deliprao · 2026-09-21
• [source] Autorubric's criterion types map 1:1 to typesafeai Jev's Noul/Score/Choice — IanArawjo · 2026-09-21
• Autorubric vs Jev: criterion types map one-to-one, real edge is parallelism — ShriramKMurthi · 2026-09-21
• Open-source rubrify compiler mirrors typesafeai's Jev abstraction, dev says autorubric predates it by 8 months — secemp9 · 2026-09-21
• Delip Rao responds to COLM paper code similarity claim: repo and tech reports public since Feb — deliprao · 2026-09-21