trainproof: a deterministic linter that tells you when a training run is already dead

CupGlass540 · reddit · 2026-08-19

After months of a non-converging 730M-param TTS model where no tool would say "this run is dead, stop paying for it," the author wrote trainproof (MIT, pip install trainproof): it reads your existing training logs and returns a verdict with an exit code — pure rules, no model judging a model.

Verdict design: FAIL→exit 1, WARN/PASS→exit 0, NOT-CHECKED→exit 2 — deliberately separating "your run failed" from "I couldn't read your log" so CI isn't silently lied to.

Fault-injection validation: one Qwen2.5-3B QLoRA, six configs (healthy, 100x LR, lr=0, fp16 NaN, shuffled labels, overfit), three seeds. The 100x LR run spiked grad-norm to 2,650 (4,900x its median). Notably, the shuffled-labels run dropped loss from 18.9 to 5.7 on unlearnable data — memorising noise statistics is indistinguishable from real training on a single curve, and is stated as a limitation in the README.

Corrected by real logs: TP-ZERO-GRAD once FAILed a healthy 125k-step Coqui run because Coqui writes avggradnorm as 0.0 when clipping is off. The fix was reasoning, not thresholds: a run cannot both learn and receive no gradient, so the check stands down when loss improved, and records why.

Reads HF trainerstate., Coqui, TensorBoard events, JSONL, CSV; the tfevents reader is written from the wire format — no tensorflow, torch, or protobuf — validated byte-exact against EventAccumulator.

Original post →

trainproof: a deterministic linter that tells you when a training run is already dead

More from Infra