GEPA Prompt Optimization Lifts Jev's F1 From 69.1% to 79.7% on Medical Literature Task
matei_zaharia · x · 2026-09-21
A hands-on guide (reshared by Matei Zaharia) tests TypeSafe AI's 'System One' decision model Jev on identifying adverse drug event sentences in public medical literature. First pass: every Jev output passed format checks, but 54 of 300 test answers disagreed with labels — near-perfect recall on positives, many false positives, and occasional fully-confident wrong answers. Connecting GEPA, a prompt optimizer that proposes instruction revisions from training errors, clarified the evidence each sentence required (drug, harmful effect, relationship). On a fresh test set, F1 rose from 69.1% to 79.7% and probability error nearly halved, at the cost of two extra missed positives. Jev, released September 15, 2026, returns choices, scores and probabilities for agent-critical-path decisions (relevance checks, model routing, tool-call review); TypeSafe claims 193.6× faster and 444.6× cheaper in selected workflow evaluations. Key lesson: valid output ≠ trustworthy output; error-driven GEPA prompt revision is an effective domain-adaptation path. Notebooks and code are included.
More from coding & agent
- HarnessRouter open-sources one API to run Codex, Claude Code and 14 agent harnesses — gaganghotra_ · 2026-09-21
- ECDYSIS paper: patch agent harnesses by failure patterns, not failure counts — rohanpaul_ai · 2026-09-21
- EU LLM apps: developer maps the 5 blockers between prototype and paid launch — felix_baron · 2026-09-21
- Coding agent UX gripe: Astra says "PR is up" without linking to it — altryne · 2026-09-21
- God's Eye View: open-source spy-satellite simulator with real data hits 39.5k GitHub stars — alex_verem · 2026-09-21
- Superlinear Episode Details the Fall 2026 Workflow for Starting Projects with Coding Agents — samgoodwin89 · 2026-09-21