JEV eval: when state evidence is ambiguous, choice order dominates and IDK fails 0/80
iamrobotbear · x · 2026-09-21
- The author replicated a JEV eval centered on whether the state sent to JEV contains enough decision-relevant evidence.
- Experiment 1: with state + intent question, choice order dominates when evidence is ambiguous—both options are supported and nothing signals which should dominate.
- Experiment 2: adding an IDK option barely helps because "which intent is primary" and "is there enough evidence" get mixed into one choice; IDK won 0/80 runs.
- Experiment 3: asking sufficiency separately makes JEV assign 38.75%-38.95% probability that the state is sufficient—split the question instead.
More from coding & agent
- Ruff author charliermarsh: he only started using agents meaningfully in December 2025 — charliermarsh · 2026-09-21
- Open-source LLaMA-Factory fine-tunes 100+ LLMs; 200 examples can beat frontier models — Roger_M_Taylor · 2026-09-21
- SWE-2 free across Devin Cloud Agents, CLI and Desktop until October 8 — silasalberti · 2026-09-21
- Delta and OpenTable block AI agents, hinting at a coming platform stand-off — Scobleizer · 2026-09-21
- ForgePoint Signal launches MCP-native daily US estate and tax law monitoring — modelcontextprotocol · 2026-09-21
- Memory MCP adds persistent semantic memory to AI assistants via Turso and OpenAI — modelcontextprotocol · 2026-09-21