JSON setup for jev judge surfaces 8 hallucinated refund sessions in 48
strickvl · x · 2026-09-24
Showing the JSON config for the TypeSafe jev evaluator: jev gave the timeline question p(yes)=0.93, failing the session. Across 48 sessions imported from the tracing provider, 8 contained hallucinated refund timelines — easy to confirm manually once surfaced.
Related event: Testing the jev Judge: 8 Hallucinated Refund Timelines Found in 48 Sessions(2 posts)→
More from coding & agent
- "This Is Too Slow, Make It Faster" — Dev's One-Liner Actually Optimized His AI Code — DanielLockyer · 2026-09-24
- SwiftFairy launches: on-device Mac app that reviews your coding agent's Swift code — JordanMorgan10 · 2026-09-24
- Stripe's Link Wallet Tops 300M Consumers, Partners With Muse for Agentic Buying — jeff_weinstein · 2026-09-24
- Zuckerberg Put Cameras in His MMA Gym So an Agent Could Coach Him Between 1-Minute Rounds — victor_explore · 2026-09-24
- LangChain's Interrupt keynote ships LangSmith Engine v2, Deep Agents 0.8, fine-tuning and more — LangChain · 2026-09-24
- Runable launches Scheduled Tasks: agents that do the work, not just remind you — SimplyAnnisa · 2026-09-24