ZenML's open-source Kitaru turns production agent traces into replay-based eval loops
strickvl · x · 2026-08-18
The ZenML team launched a major update to Kitaru, an open-source framework for replay-based AI agent evals.
The premise: thousands of agent traces sitting in production are doing nothing. Kitaru ingests traces from Langfuse, Braintrust, or any OTel source, investigates what went right or wrong, groups recurring failures into cohorts, and builds evaluators around them. You can then replay those same real cases against a different model, prompt, context, or agent setup and see what would have changed.
The flow: wrap an existing agent with KitaruAgent, import traces, have your coding agent read 20 runs and define cohorts, then run comparison experiments (the demo shows 90/90 failures dropping to 4/90). The same experiment can run on every commit to guard against regressions. Free and open source.
More from coding & agent
- Sentence Transformers v6.0 Ships Late Interaction Models, Its Largest Update Yet — tomaarsen · 2026-08-18
- sentence-transformers v6.0 adds ColBERT-style multi-vector late interaction retrieval — tomaarsen · 2026-08-18
- Sentence Transformers v6.0: fp16 + FlashAttention Hits 3.87x Over fp32 — tomaarsen · 2026-08-18
- Sentence Transformers v6.0 Arrives: Late Interaction Lands in Project's Biggest Release — tomaarsen · 2026-08-18
- Breaking Changes in Sentence Transformers v6.0: transformers v5 Floor, API Shifts — tomaarsen · 2026-08-18
- bfloat16 Sigmoid Saturates CrossEncoder Rankings; Fix Lifts nDCG From 0.18 to 0.68 — tomaarsen · 2026-08-18