Detecting Performance Regressions Using ML and Hardware Counters

ZeroDark_Hereford · reddit · 2026-08-14

The author discusses engineering practices for using machine learning and anomaly detection in performance regression testing.

Current Setup:

Healthy runs are used to learn normal behavior, while regression runs test anomaly detection.
Only about 10 healthy samples exist per counter group.
Leave-one-out is used on healthy data to set detection thresholds; regression samples are excluded from training/thresholding.

Key Questions:

Is a standard train/validation/test split still necessary for one-class anomaly detection?
With only 10 healthy samples, is leave-one-out better than a percentage-based split?
Can regression samples serve directly as the unseen test set?
Should evaluation focus on false-positive rate and recall rather than MSE/MAE?

More from Infra