How Numan built a compliant AI health coach on LangGraph, where eval criteria mattered most
LangChain · x · 2026-10-20
UK digital health company Numan explains how it built Nu, an AI health coach for weight, hormone and sexual health programs, in a heavily regulated medical space where agents must not diagnose, prescribe, or make clinical decisions.
- The team started with the standard playbook: static input/output fixtures scored by an LLM judge — and it failed, because coaching is multi-turn conversation with no single right answer, and safety depends on conversational context.
- Single-turn evals missed key behaviors like understanding patient blockers across turns and holding boundaries under pressure; averaging scores across datasets hid failure modes.
- Key lesson: the quality of evaluation criteria, not the agent itself, mattered most for shipping fast without crossing the medical-device line. Stack: LangGraph + LangSmith.
Related event: Numan Builds Regulated AI Health Coach on LangGraph with Evaluation Loop(4 posts)→
More from coding & agent
- Index Network open-sources Social Discovery Protocol: agents negotiate fit on your behalf — carsonfarmer · 2026-10-23
- Installed a formal-verification skill: it silently installed Java, MacBook runs 10 degrees hotter — zealcaiden · 2026-10-23
- OpenAI's Decisions API hits GA: near real-time routing up to 10x faster than GPT-6 Luna — OpenAIDevs · 2026-10-23
- Ruff author warns: upgrade Node/Bun now — --use-system-ca still accepts the eMudhra cert — charliermarsh · 2026-10-23
- Proof of Agent: paste one prompt and your AI mines, grades, and self-governs — adamamcbride · 2026-10-23
- DeepLearning.AI weekly: GPT-6.1 hits 97.9% byte parity on COBOL migration — DeepLearningAI · 2026-10-23