Rule-based checks scored 1.00 while the agent hallucinated a refund timeline
strickvl · x · 2026-09-24
A demo customer service agent refunded a damaged order and told the customer to allow '3-5 business days' — but none of its tools mention refund timing, so it was invented. Existing rule-based checks still scored the session a perfect 1.00 because rules can't read replies. That gap is exactly what the new jev judge evaluator addresses.
Related event: Kitaru Integrates jev Judge to Cheaply Catch Agent Hallucinations(2 posts)→
More from coding & agent
- "This Is Too Slow, Make It Faster" — Dev's One-Liner Actually Optimized His AI Code — DanielLockyer · 2026-09-24
- SwiftFairy launches: on-device Mac app that reviews your coding agent's Swift code — JordanMorgan10 · 2026-09-24
- Stripe's Link Wallet Tops 300M Consumers, Partners With Muse for Agentic Buying — jeff_weinstein · 2026-09-24
- Zuckerberg Put Cameras in His MMA Gym So an Agent Could Coach Him Between 1-Minute Rounds — victor_explore · 2026-09-24
- LangChain's Interrupt keynote ships LangSmith Engine v2, Deep Agents 0.8, fine-tuning and more — LangChain · 2026-09-24
- Runable launches Scheduled Tasks: agents that do the work, not just remind you — SimplyAnnisa · 2026-09-24