Open-source Orinyx tests healthcare AI agents against FDA labels, no LLM in the verdict
Practical_Green5775 · reddit · 2026-11-14
The team behind open-source verification project Orinyx argues that a correct answer doesn't make an agent reliable—slightly changed inputs, missing information, or confident wrong answers need systematic testing. The project currently focuses on healthcare and medication-related AI workflows.
Key pieces:
- Independent evaluation of AI outputs, failure identification, and evidence behind verdicts
- Orinyx API for programmatic verification integration
- Orinyx Arena, a shared environment to compare AI systems on identical scenarios with measurable results
- The orinyx-mcp GitHub repo is the official MCP server: it tests healthcare AI agents against cited FDA labels, with deterministic verdicts and no LLM in the loop
Still early, they're soliciting developer feedback on agent testing practices beyond checking expected answers.
More from coding & agent
- Money-saving trick: let a cheap model do the work, then ask it to summon a pricier model to review — vista8 · 2026-11-14
- Open-source personal agent The Machine drafts everything but never sends without your OK — markloperman · 2026-11-14
- I tested 1,682 public MCP servers: 51% have tool inputs with no description — abhiramreddykalluru · 2026-11-14
- Solo AI system sold for $18,880 watches 1,300 execs and surfaces $50k–$200k deals — aryanXmahajan · 2026-11-14
- AAA game dev builds MAC, a multi-agent council where GPT, Claude and Codex review each other — Hirox-rd · 2026-11-14
- Perplexity team teases minimalist LLM trainer built with coding agents, open source soon — BanghuaZ · 2026-11-14