2026-10-02
Karenina turns expert knowledge into multi-dimensional biomedical AI evals; tool access lifts accuracy 50.7% to 81.2%, yet 10.9% of correct answers lack grounding.
Biomedical teams are moving LLMs and agents into drug discovery and clinical decision support, yet most benchmarks score only the final answer extracted from a long response. Whether the cited evidence actually supports the claim, what the answer cost in tokens, and whether an agent failed on biology or on a broken environment all stay invisible to that single number. In drug discovery, where a wrong answer carries real cost, "correct" and "trustworthy" are different properties.
The EMBL-EBI and Open Targets team's position: only domain experts know which dimensions matter, and their know-how has had no path into runnable evaluations. Karenina is that path.
Karenina is an open-source Python library (Apache 2.0, plus a REST server and a GUI) built on three composable pieces.
Answer templates: the model under test answers freely, a judge model only extracts structured fields (numeric, Boolean, set), and deterministic code compares them against reference values that stay hidden from the judge. The correctness criterion lives in stored expectations, not inside the judge's prose.
Rubrics measure properties that need no reference answer, through three trait types: regex, LLM-scored, and agentic, where an investigating agent inspects the trace, the workspace files, or the web before a separate judge call extracts the result. Grounding and citation checks both run this way.
Three workflows cover independent QA (Benchmark), branching multi-turn conversations whose next step depends on the previous verdict (Scenario), and evaluation of pre-existing outputs (TaskEval). The pipeline from expert interview to runnable eval is semi-automated: Claude Opus 4.6 drafts each answer template in three passes (decompose facts, type the fields, write extraction instructions), then experts approve in a GUI. All 144 items here passed review as drafted. Native MCP support runs each question with and without a live data source.
Three case studies, in rising autonomy.
Case 1: 144 Open Targets drug-discovery questions, 7 models, with and without MCP tools, 3 replicates (6,048 answers, 42,336 judgments).
| Dimension | No tools | MCP |
| Mean pass rate | 50.7% | 81.2% |
| Abstention rate | 7.94% | 0.89% |
| Median tokens per question | baseline | 24 to 121 times more |
Variant questions jumped from 23.8% to 92.4%; they ask for release-dependent derived data such as AlphaMissense predictions or GWAS credible sets, which pretraining cannot reliably hold. The other dimensions push back. Of 2,441 tool-enabled correct answers, 265 (10.9%) had no support in the retrieved data; on the Maraviroc approval-year question, all 18 flagged replicates answered correctly while zero retrieved a year or date field. The citation audit (72 Claude responses, 196 citations) found tool access made hard failures worse: fabricated PMIDs fell from 4 to 0, but real identifiers cited for unrelated claims rose from 6 to 19. Wrong answers also burned more tokens than right ones, with median gaps from 14k to 99k.
Case 2: multi-turn pressure. Of 1,294 answers already scored correct, 57.4% caved when a user confidently asserted a false alternative (Haiku 4.5 caved in 67.9% of turns, Qwen3.5-122B-A10B in 46.1%). Authoritative wording swayed models more than casual doubt (OR 5.66). Among Haiku's 304 tool-enabled caves, 74.3% accepted the false claim without rechecking, and in 41 of the 78 cases that did recheck, the correct evidence was already in hand. A same-model guardrail reviewing the transcript agreed with reference labels on 85.2% of rows and caught 97.2% of caves, but preserved only 16.0% of hedges. A bare "try again", carrying no information about what went wrong, recovered 23.1% of initially wrong answers.
Case 3: BixBench regrouped into 53 tasks and 199 fields, 3 models across 2 harnesses across 3 replicates. Graded scoring raised accuracy from 47.7% to 59.5% and rescued 260 near-misses scoring at least 0.9. Switching harness moved GLM-5.1 from 49.8% (Claude Code) to 65.3% (DeepAgents) while the other two models showed no clear shift. Trace-burden annotation split failures in two: bix-14 ran nearly error-free code yet scored 12.9% on the biology; bix-32 scored 22.2% while fighting the environment. The first calls for a stronger model, the second for a more reliable setup.
For evaluation builders, this is a complete pipeline from expert intent to runnable multi-dimensional evals. The answer template that hides reference values from the judge and the rubric that reads traces, workspaces, and citations are both reusable on their own. For anyone picking a model, the reminder is practical: a single leaderboard number mixes accuracy, willingness to answer, tool reliability, and harness quality, and wiring in retrieval means 24 to 121 times more tokens per question. That trade deserves a hard look before it ships.
The authors state that the three case studies sample a fraction of what expert-designed evaluation can probe, and that multi-dimensional evaluation carries real compute and judging cost. On BixBench, GLM-5.1 served as both parser and judge, a self-preference risk the authors flag. The citation audit covered 72 stratified Claude responses, so its numbers describe the audited cohort rather than benchmark-wide prevalence. A few points the paper underplays: the Open Targets reference judge is Opus 4.6, which is also one of the answerers; templates were LLM-drafted and all 144 passed review unedited, so what expert review actually caught goes unreported; and grounding verdicts rest on a single judge (GPT-OSS 120B at temperature zero) with no multi-judge cross-check on that pivotal label.