Reading a Pretraining Run: A P0/P1/P2 Metric System for Monitoring LLM Pretraining
SonglinYang4 · x · 2026-09-19
Chunyuan Deng published "Reading a Pretraining Run," a systematic guide to the metrics used for monitoring large-scale LLM pretraining, with definitions, default cadences, and alert priorities.
- Probes and priority tiers: A large run emits tens of thousands of probes; the hard part is knowing which ten to watch. Metrics are tiered as P0 (wired to on-call, ten metrics, each trigger stops the run or wakes someone, sampled every step), P1 (diagnostics opened when a P0 moves, per-layer/per-tensor, every 100 steps), and P2 (forensic detail per-parameter/expert/domain, on demand).
- The nine P0 first-screen signals: global and worst-rank loss means (catching convergence issues or a single worker/shard carrying damage), per-source and per-modality loss (catching domain regression under a flat global curve), global gradient norm and clip rate (the earliest numerical warning, moving before loss), co-spike count of loss and gradients on the same step, step time and throughput (stalls, stragglers, communication regressions), peak memory maxed over ranks (OOM headroom), and expert load MaxVio (routing collapse and compute imbalance).
More from Infra
- GPU host warns: renter exploited his rig for attacks, Clore.AI blocked him for reporting it — anomaly256 · 2026-09-19
- 3M paid $10.3B to quit PFAS — AI data centers just made it a growth market again — aakashgupta · 2026-09-19
- $250 of modded mining cards, 30GB VRAM: old i7 PC runs Qwen at 30 tok/s with patched drivers — HFq_Dev · 2026-09-19
- Dual RTX 5060 Ti Only Gets 10 t/s on Qwen3.8-Flash-Next, Seeking Config Advice — MkGod · 2026-09-19
- Hacking open-source model behavior with sglang's scoring endpoint, no fine-tuning needed — BLUECOW009 · 2026-09-19
- Distilling DeepSeek V4 Flash to a 4B model on DGX Spark: 26 hours, 22ms per judgment — Dan_Jeffries1 · 2026-09-19