ZenML open-sources Kitaru: replay production agent traces against different models or prompts
htahir1 · reddit · 2026-08-18
ZenML open-sourced Kitaru (Apache 2.0, fully self-hostable) to answer: given a real production agent session, what would have happened if you changed the model, prompt, context, or tool policy — without calling live tools again.
Key mechanics:
- Imports traces from Langfuse, LangSmith, Braintrust, Logfire, JSONL, or OpenTelemetry into investigable, replayable sessions;
- During replay your actual agent code runs, but tool calls are answered from the original recording (a recorded refund result is returned instead of issuing a new refund);
- Replay the agent unchanged first to establish a baseline; if the reproduction isn't close enough to the original run, counterfactual runs shouldn't be trusted either. Then change one variable and replay.
The broader workflow: production traces → investigate → cohorts → expert judgment → evaluators → replay → experiments. Kitaru proposes cohorts of similar sessions from recurring patterns; expert reviews calibrate evaluators, which are Python functions versioned alongside the agent.
The maintainer seeks feedback on two questions: how to handle tool calls during replay, and what replay fidelity is required before trusting a counterfactual.
Related event: ZenML Open-Sources Kitaru for Replay-Based Agent Evaluation(2 posts)→
More from coding & agent
- Open-source Meetily hits 29k stars: fully local AI meeting transcription and summaries — tom_doerr · 2026-08-18
- Gemini 3.7 Flash 2-6x Faster in Planning, Recommended Workflow Setup — rakyll · 2026-08-18
- LlamaParse Now Handles Revision Tracking for Clean Redline Document Parsing — llama_index · 2026-08-18
- LangChain Panel: Automating Eval & Environment Engineering — LangChain · 2026-08-18
- Nehemiah: On-demand Linux microVMs to hand to an AI — Rasmic · 2026-08-18
- Building Zero-Trust AI Agents: Google ADK Guide to Prevent Prompt Injection — rseroter · 2026-08-18