CallBaba launches BabaBench, a benchmark testing AI agents on real patient advocacy tasks
mattturck · x · 2026-10-22
Healthcare coordination startup CallBaba released BabaBench, a simulation benchmark measuring frontier AI agents on real long-horizon patient advocacy tasks.
- Tasks rebuilt from real cases: finding subsidized housing, checking eligibility and waiting lists, meeting deadlines, coordinating housing authorities, insurers and navigators.
- The team argues chatbots aren't enough — agents must call multiple parties, request consent and submit applications, and "completing the task is possible; doing it safely is the hard part."
- Evaluation goes beyond completion to check whether agents acted safely.
Connor Sweeney discussed on the Jordo & Coogan show why they prioritized humans-in-the-loop over another AI personal assistant, and says substantial RL work lies ahead.
More from coding & agent
- Microsoft Research's PDD Kit formally proves AI-generated code meets your intent — omojumiller · 2026-10-22
- d3-parsets v3.0.0 rewritten for D3 v7, visualizing how people use Claude — Elijah_Meeks · 2026-10-22
- SF Systems Meetup: Agents-Only Version Control and Postgres Rewritten in Rust — ShadajL · 2026-10-22
- BitKindle: AI bootstraps a full C compiler from a 3,584-byte machine-code seed — dotey · 2026-10-22
- dsh-prompt-layer plugin injects your rules into DSH's system prompt, overriding AGENTS.md — teortaxesTex · 2026-10-22
- LLM writing still garbage in 2026, dev shows the blunt prompt he uses — arthurcolle · 2026-10-22