OncologyBench: best agents pass only 29-40% of verifiable multistep cancer decision tasks
kenbwork · x · 2026-10-14
Latch introduced OncologyBench, a verifiable benchmark of 25 multistep oncology evaluations (built on 92 component tasks) spanning six stages of the cancer journey, using published datasets, case studies and patient-donated data.
Key findings:
- Across 31 model-harness combos, Claude Opus 5, Opus 5.5 and Sonnet 5.5 led at 29-40% pass rates; Kimi K3 hit 24%.
- Models with strong analysis skills still fail at reasoning: two-thirds of attempts that got every analysis right still failed the final decision.
- Decision quality varied with the questioner's persona.
Example: a five-specialist-agent workup of a stage II mucinous cecal cancer patient where the correct call is observation, not adjuvant chemo (MSI-high, ambiguous DPYD2A reads).
Related event: OncologyBench Released: Top Models Pass Only 29-40% of Oncology Decisions(7 posts)→
More from AGI Musings
- a16z and Dan Boneh: AI's math leap could compress a decade of crypto trust-building into one month — a16z · 2026-10-14
- Who bears the burden of proof on AI doom? Accelerationists and doomers clash — AccBalanced · 2026-10-14
- MIT PhD Student Co-Authors Book Chronicling the Rise and Fall of the Tech Worker Movement — soumitrashukla9 · 2026-10-14
- Tim Sweeney asked if 5-minute AAA game generation would be good for humanity, sparks debate — max_paperclips · 2026-10-14
- MIT professor's ten predictions for AI for science, from executable papers to compiled matter — ProfBuehlerMIT · 2026-10-14
- Sweden spends hundreds of millions of kronor on printed textbooks as 2.8M-child study links screens to lower grades — Eric520CC · 2026-10-14