Typed Evals: open-source framework for calibrated Jev-powered LLM evaluation
Charming_Group_2950 · reddit · 2026-09-21
Inspired by the recent popularity of "System One" judge models like Jev, the author open-sourced Typed Evals, a Python framework for evaluating LLMs, RAG pipelines, and AI agents with a focus on calibration:
- Supports LLM response, RAG, and agent/tool-trace evaluation, plus human-label calibration
- Fits per-metric calibration curves against human pass/fail labels, so a raw Jev score of 0.8 maps to actual human acceptance rates instead of arbitrary thresholds
- Flow: human labels → Jev scores → per-metric calibration → held-out validation → reuse calibrated evaluator
- Async/batch evaluation, custom judge backends, integrations with LangChain, CrewAI, Microsoft Agent Framework; core stays framework-agnostic
The project is early-stage and public on GitHub (TrustifAI/typedevals); the author is seeking feedback.
More from coding & agent
- Hermes Agent Now Streams Its Reasoning Live Into Any OpenAI-Compatible App — Teknium · 2026-09-21
- Devs clash over whether beginners should still learn to code as AI automates code review — mgill25 · 2026-09-21
- Pair vibecoding with a novice reveals the real unlock: communication and systems thinking — eschadiol · 2026-09-21
- Substrate bets the agent loop is the new scheduling loop with fast suspend/resume — rakyll · 2026-09-21
- Long-running coding agents let you delegate work all weekend — josh_bickett · 2026-09-21
- Many Mac mini buyers would pay ~$1k for always-on cloud Codex/Claude Code, says alliekmiller — alliekmiller · 2026-09-21