Open Source Trainproof: A Deterministic Linter for ML Training Runs

CupGlass540 · reddit · 2026-08-05

A developer open-sourced trainproof (MIT, pip install trainproof) after struggling with non-converging 730M-parameter TTS training. It's a deterministic linter for ML training runs that reads existing logs and returns a verdict with an exit code, avoiding ML-judging-ML hallucinations.

Core Design & Validation

Rule-driven: All checks are deterministic rules that fire or don't. Severity and exit codes are strictly separated (FAIL=exit 1, WARN=exit 0, NOT-CHECKED=exit 2).
Fault Injection Testing: Validated using Qwen2.5-3B QLoRA across 6 configurations (healthy, 100x LR, lr=0, etc.), catching anomalies in seconds.
Handling Limitations: Discovered that shuffled labels still reduce loss (memorizing noise). The tool introduces a compare feature to expose this by comparing against a known-good baseline's relative floor.

Lifecycle Coverage

Pre-GPU: Dataset + tokenizer lint, checks entrypoint imports, checkpoint integrity, RAM, and disk.
During: One-line HF callback; can abort diverging runs if opted in.
Post-run: Detects divergence, flatlines, NaNs, and overfitting (e.g., an XTTS v2 run of 125k steps was correctly flagged as FAIL).
Vs Baseline: Relative-floor rules.

It reads HF trainerstate.json, Coqui, TensorBoard event files, JSONL, and CSV. The tfevents reader parses from the wire format without requiring tensorflow or torch.

Original post →

Open Source Trainproof: A Deterministic Linter for ML Training Runs

More from coding & agent