Developer open sources NoiseCheck, exposing statistical fallacies in model evals
Formal-King3851 · reddit · 2026-08-16
The author built a CLI tool called noisecheck to determine if model evaluation improvements are real or just statistical noise. Real-world tests on models like DeepSeek and GLM revealed that a "lead of 2.7 points" can be statistically meaningless, with rankings flipping upon scaling the sample size. Experiments also showed that a single model's score fluctuated by ±2.9 points across identical runs, exceeding the marginal gaps between different models.
The tool features cluster-aware resampling, judge consistency checks (GPT-4 vs. human experts Kappa 0.15), and power analysis, designed to act as a CI gate to prevent releases based on spurious data. The code is open source.
More from coding & agent
- atuin v18.24 adds cross-device sync to resume AI threads across Claude and Codex — braelyn_ai · 2026-10-16
- Agents turn conservative as context grows: edits shrink from 150 to 30 lines — dbreunig · 2026-10-16
- ai& becomes an official built-in Cline provider with 13 open-weight models — DavidBennett__ · 2026-10-16
- Open-source rizin-mcp lets LLMs run local binary and malware analysis via Rizin — Pure_Emergency1424 · 2026-10-16
- x402 as a free GTM channel: agents discover 100+ tools, no API keys needed — 0xJeff · 2026-10-16
- Sam Altman: stop getting better at prompts, build loops and graphs that prompt for you — goyalshaliniuk · 2026-10-16