Don't just watch training loss: a practical checklist for reading ML training logs

cneuralnetwork · x · 2026-11-08

The author shares how to systematically read an ML training log instead of fixating on training loss:

Training loss dropping: good, but not the whole story
Validation loss rising: likely overfitting (example chart shows valloss climbing from 2.01 to 2.31)
Gradient norm spiking: investigate instability (gradnorm jumps from 1.4 to 12.3)
Throughput dropping: check GPU utilization, data loading, sequence lengths (tok/s falls from 42k to 28k)
LR decreasing: expected with a decay schedule

An attached example log table shows how to spot trouble via lr/gradnorm/tok/s/valloss even when loss looks healthy.

More from Research