Is your model swap actually worse, or just noise? Open-source Assay brings stats to LLM evals
Professional-Run3614 · reddit · 2026-10-13
While testing local models and quantizations for legal document processing, the author hit flaky evals. His fix: run each case multiple times on both versions and compare pass rates with multiple-comparison correction — 8/8→7/8 is likely noise, 8/8→0/8 a clear regression. Two key findings: the LLM judge is sometimes the flaky component (judge the same answer repeatedly to detect it), and timeouts/judge errors shouldn't count as failures. He open-sourced this as Assay, runnable via pytest --assay --repeat 8 with any OpenAI-compatible endpoint.
More from coding & agent
- Notion designer argues against personal agents: the future of work is a factory — nbaschez · 2026-10-13
- clauden: run Claude Code on free NVIDIA-hosted models via a LiteLLM proxy — gmaggiotti · 2026-10-13
- Anthropic Details Flows at AI Engineer NYC: Unix-Inspired Orchestration for Claude Agents — brada · 2026-10-13
- ux-ui-agent-skills turns Claude into a design architect with 52 components and 50 hard quality gates — tom_doerr · 2026-10-13
- DSPy 3.4.1 ships GEPA structured proposer as default and RLM sub-agents — dbreunig · 2026-10-13
- Yacine: all software is trash, so with AI it's now reasonable to just build it yourself — yacineMTB · 2026-10-13