Open-source LLM regression gate digline hits 1.0: quantifying judge noise before calling a regression
alexpran · reddit · 2026-11-02
Developer alexpran shipped digline, an open-source gate that compares an LLM app against an approved reference and measures the judge's own noise before flagging a regression, reaching 1.0 just 67 days after 0.1.0.
- 1.0 shipped with the planned schema rework still undone, explicitly documented in a changelog section titled "What 1.0.0 does not contain, and why it can wait".
- 1.0 is framed as a compatibility promise, not a feature list: minors won't break public Python signatures, and the wire JSON and stored documents are versioned independently.
- Along the way the author found a real bug of the kind the tool exists for: the ADR's rule for minimum sample agreement was wrong for even n (a 2-2 split with 4 samples and a 3/4 floor should be an error) — nothing tested that sentence; 69 cases now do.
- The author asks fellow tool maintainers when they called 1.0 and whether it changed who tried it.
More from coding & agent
- Google's open-sourced "Elephant-Goldfish" workflow separates planning from review to curb sloppy AI output — gaganghotra_ · 2026-11-02
- InfoQ: A Systemic Approach to Memory, Knowledge, and Context in Agentic AI Architectures — AdiPolak · 2026-11-02
- Autonomous's Harness Pro uses an ESP32-P4, not a mini PC, to monitor AI coding agents — dee_hw · 2026-11-02
- RL-trained LFM2.5-24B adapter matches accuracy with 20% fewer tool calls — yb2698 · 2026-11-02
- Debugging production AI agents: error surfaces at step 5, root cause at step 3 — Ok-Machine7983 · 2026-11-02
- RAGFlow 1.0 preview: from RAG engine to agent-native database — newpeak · 2026-11-02