8 of 48 agent sessions had hallucinated refund timelines, caught by jev judge
strickvl · x · 2026-09-24
To test whether TypeSafe's jev judge was just number-matching, strickvl edited one field at a time in a session: a '30 days' from the policy tool passes, but promising a customer 30 days fails — even though a 30 appears in the tool results (the return window, not the refund time), showing real semantic understanding.
In practice, jev gave the timeline question p(yes)=0.93; across 48 sessions imported from their tracing provider, 8 contained hallucinated refund timelines, easy to confirm manually once surfaced.
Related event: Testing the jev Judge: 8 Hallucinated Refund Timelines Found in 48 Sessions(2 posts)→
More from coding & agent
- "This Is Too Slow, Make It Faster" — Dev's One-Liner Actually Optimized His AI Code — DanielLockyer · 2026-09-24
- SwiftFairy launches: on-device Mac app that reviews your coding agent's Swift code — JordanMorgan10 · 2026-09-24
- Stripe's Link Wallet Tops 300M Consumers, Partners With Muse for Agentic Buying — jeff_weinstein · 2026-09-24
- Zuckerberg Put Cameras in His MMA Gym So an Agent Could Coach Him Between 1-Minute Rounds — victor_explore · 2026-09-24
- LangChain's Interrupt keynote ships LangSmith Engine v2, Deep Agents 0.8, fine-tuning and more — LangChain · 2026-09-24
- Runable launches Scheduled Tasks: agents that do the work, not just remind you — SimplyAnnisa · 2026-09-24