llm-drift-bench: open-source tool diffs LLM outputs to catch regressions when switching models
Worldly-Knowledge-90 · reddit · 2026-11-13
A developer released llm-drift-bench 0.1.0, a Python tool that compares responses from your old model against a candidate model and flags missing JSON fields, changed numbers, and text differences — so you can see exactly what breaks when switching models or quantizations.
It works on saved responses, no API key or GPU needed. The author is collecting real-world failure examples for 0.2.0.
Related event: Open-source llm-drift-bench compares outputs before and after model swaps(2 posts)→
More from coding & agent
- LangChain raises $300M at $3B valuation, with 7,000+ companies on LangSmith — LangChain · 2026-11-14
- Largest systematic agentic eval campaign launches: test agents as tool users, not tools — animesh_garg · 2026-11-14
- Perplexity agent orchestrated 48 parallel sandboxes to render photoreal Pantheon video in 3 hours — AravSrinivas · 2026-11-14
- Meta Ships Edits Desktop: SAM-Powered Cutouts Plus an MCP for AI Agents — nikhilaravi · 2026-11-14
- Cat Wu, ex-Claude Code product lead now at OpenAI: 10 great evals beat hundreds — aakashgupta · 2026-11-13
- AsyncGRPOTrainer now stable in TRL v1.18.0: generation and training on separate GPUs — SergioPaniego · 2026-11-13