How real teams run LLM evals in production: tools kept, tools dropped
LostLeek1304 · reddit · 2026-10-24
A developer building customs-entry checks (Norquin) asks how teams with LLMs in real production run evals day to day:
- Do evals run pre-release, on live traffic, or both? How are CI gates, sampled human review, LLM-as-judge, and drift alerts combined?
- Which tools stayed (Braintrust, Langfuse, LangSmith, Phoenix, Promptfoo) and which were dropped?
- Their own setup: a golden test entry matching to the cent, a deterministic gate rejecting parses where line totals don't add up, and Telegram alerts on failures — no tracing, no platform
- Cautionary tale: the golden test passed 38/40 lines, but two real documents scored 0/38 because the test file had a column real ones don't print
More from coding & agent
- ClimateTriage MCP Server lets AI agents search open-source climate issues — modelcontextprotocol · 2026-10-24
- NoonAI DIS launches verified MCP for automatic face detection and de-identification — modelcontextprotocol · 2026-10-24
- Dev adds a 'Non-goals' section to agent work orders to stop unsolicited refactors — HaktanSuren · 2026-10-24
- How do you turn a real agent workflow into a useful evaluation set? — Super_Tough_4997 · 2026-10-24
- Glance ships MCP server so AI agents can create and update native iPhone Home Screen widgets — Dense-Map-406 · 2026-10-24
- Microsoft open-sources Agensh, a decentralized multi-agent harness scaling to 1,024 agents — donglixp · 2026-10-24