Closing the loop from eval failure to prompt fix
Future_AGI · reddit · 2026-07-20
The post argues that most LLM eval/observability stacks stop at detection: they catch a bad answer, but leave the actual fix, redeploy, and regression-proofing to the user.
It reviews several tools:
- LangSmith: strong tracing and evals; scoring happens after the response; self-hosting is Enterprise-only.
- Weights & Biases Weave: tracing plus inline guardrails that can block bad output before it reaches the user; self-hosting is enterprise.
- Arize Phoenix: OpenTelemetry-native tracing/evals with a large community; source-available under Elastic License.
- Braintrust: built around turning production misses into tests and rerunning them after prompt/model changes; includes Loop for prompt rewriting; production scoring is post-response.
- Galileo: Protect can block inline; its Agent Control policy layer is open-sourced under Apache-2.0, while eval/observability remain enterprise.
- Comet Opik: Apache-2.0 and self-hostable, with guardrails and an Agent Optimizer; its gateway is still beta and does not yet control tool/MCP access per run.
Their own open-source platform claims to close the loop end to end on a single trace: tracing, eval scoring, inline refusal, and prompt optimization all feed each other. It runs on Docker Compose, is Apache-2.0, keeps prompts/outputs on hardware you control, and can govern which tools or MCP servers an agent may call on each run.
The author ends by asking whether readers only treat eval failures as dashboard/regression issues, or actually block and fix them before shipping.
More from coding & agent
- Cognition's SWE-2 uses a KKT duality argument in RL to shift the effort Pareto curve — YouJiacheng · 2026-09-11
- First-ever Three.js Conference lands in Paris, with a panel on AI-shortened design workflows — OdinLovis · 2026-09-11
- Data engineering, not agent frameworks, is the real bottleneck for enterprise AI agents — dhruv2038 · 2026-09-11
- RTK Terminal Compression Cuts Tokens but Leaves Your AI Coding Bill Unchanged — Bartaseth · 2026-09-11
- GPT-6 Astra beats Factorio with enemies in 44 in-game hours at ~$4,500 API cost — liminal_bardo · 2026-09-11
- Investment Analyst Asks How to Build a Claude-Based Diligence Agent Stack — Careless_Tie2286 · 2026-09-11