Stop chasing phantom LLM eval swings: bootstrap confidence intervals in 20 lines of Python
bgoncalves · x · 2026-09-23
Core idea
You ran the same eval twice and got 84.2, then 81.9. Without error bars, every score is a coin flip dressed as a fact — teams chase phantom regressions and burn weeks on noise.
The method
Bruno Gonçalves (author of the Data For Science newsletter, former NYU Data Science fellow) will teach in a free 30-minute live session (recorded):
- Bootstrap confidence intervals for any metric — accuracy, cost, latency — by resampling a thousand times in 20 lines of Python, no distribution assumptions
- Why identical runs swing scores: temperature, sampling, and judge variance
- Turn single scores into defensible ranges so reports survive scrutiny
More from coding & agent
- 15-year ads veteran builds the ad-platform MCP he couldn't find: 14 sources, paused-by-default writes — DapperManagement1306 · 2026-09-23
- Agent developer spends 90% of his time on planning, citing the Brooklyn Bridge engineer — doodlestein · 2026-09-23
- Tim Dettmers releases CliffCompaction, a two-command context compaction tool for Codex and Claude Code — Tim_Dettmers · 2026-09-23
- CliffCompaction installs in two commands; DeepSeek v4.1 compacts best, says Dettmers — Tim_Dettmers · 2026-09-23
- Dettmers: CliffCompaction works much better with full thinking traces — Tim_Dettmers · 2026-09-23
- With CliffCompaction, open-weight models beat closed ones in long-horizon sessions — Tim_Dettmers · 2026-09-23