FULL STORY
jev Judge Model: From Kitaru Integration to Real-World Testing
Kitaru integrated TypeSafe's jev judge model for scoring agent sessions, and community testing on 48 modified conversations caught 8 hallucinated refund timelines.
2026-09-24 ~ 2026-09-24 · 2 episodes · 4 posts
Episode 1 · Kitaru Integrates jev Judge to Cheaply Catch Agent Hallucinations (2026-09-24, 2 posts)
Kitaru now integrates TypeSafe's jev judge to score imported agent trajectories cheaply; a demo showed it catching a support agent fabricating a refund timeline that rule-based checks scored a perfect 1.00.
- Kitaru integrates TypeSafe's jev evaluator to score agent traces alongside deterministic checks — strickvl · 2026-09-24
- Rule-based checks scored 1.00 while the agent hallucinated a refund timeline — strickvl · 2026-09-24
Episode 2 · Testing the jev Judge: 8 Hallucinated Refund Timelines Found in 48 Sessions (2026-09-24, 2 posts)
strickvl tested the jev judge across 48 agent sessions, catching 8 hallucinated refund timelines, and demonstrated JSON-based configuration with jev assigning p(yes)=0.93 to timeline violations.
- JSON setup for jev judge surfaces 8 hallucinated refund sessions in 48 — strickvl · 2026-09-24
- 8 of 48 agent sessions had hallucinated refund timelines, caught by jev judge — strickvl · 2026-09-24