Ramen open-sources eval gates and guardrails for shipping MCP tools like code
Ok_Plum3595 · reddit · 2026-11-18
Ramen (BSD-3, self-hosted) treats MCP tools like code with a release process: drop a promptfoo eval config in the repo so every deploy runs against a canary and a failure blocks rollout ("no eval, no deploy"); NVIDIA SkillSpector scans repo skills plus one synthesized from what the zone actually serves; NeMo rails (selfcheckinput, jailbreakdetection, topical, hallucination) run per-tool through a judge proxy so provider keys stay on the console; client configs (CLAUDE.md block, Cursor rule, settings) are versioned and every call must carry the accepted version. Supports MCP 2026-07-28: tasks, multi-round-trip/elicitation, MCP Apps, Skills over MCP. Built largely with Claude Code agents.
Related event: Ramen: Open-Source Team MCP Platform with Eval Gating(2 posts)→
More from coding & agent
- Ant Group's Repo-Premium shows BM25 matches dense retrieval in fair code-search tests — _reachsumit · 2026-11-18
- Scoble Tests Hy4 Agent in WorkBuddy on a 20-Step Real Office Job — Scobleizer · 2026-11-18
- DetouR follows hyperlinks for multi-hop QA, adding 4.0 EM on 2Wiki with zero extra calls — _reachsumit · 2026-11-18
- Echoes fool RL search agents: 8 paraphrased false claims overturn 58-73% of correct answers — _reachsumit · 2026-11-18
- rust-anything rewrites legacy projects to Rust with AI, up to 20x faster — aigclink · 2026-11-18
- Jetriever ranks agent memories by utility not similarity, hitting gold in 84.8% of cases — _reachsumit · 2026-11-18