GPU Thermal Throttling Is Why Some of Your Training Runs Are Mysteriously Slower
dejanseo · x · 2026-10-24
@dejanseo discovered why some training runs were slower "for no apparent reason": thermodynamics. His dashboard showed the GPU at 88°C with thermal throttling likely — SM clock cut to 1515 MHz (49% of the 3105 max), power draw only 120W at 100% utilization.
The card was downclocking itself under heat, capping throughput at 123K tokens/s and 316ms per step. Lesson for anyone training on consumer GPUs: watch temps and SM clocks as closely as loss curves.
More from Infra
- Meta's Muse Spark Open hits ~150M tokens/s per MW on NVIDIA GB300 in early benchmarks — ArtificialAnlys · 2026-10-24
- Musk says no GB300s or even H100s to rent; analyst calls China chip export policy absurd — Gregory_C_Allen · 2026-10-24
- Polymarket odds: 25% chance of another AWS service disruption by end of month — Polymarket · 2026-10-24
- Amazon CEO Andy Jassy: AWS cloud capacity sold out through 2027 amid AI demand — Polymarket · 2026-10-24
- Berth parks idle AI engines in RAM, cutting model swaps from 13.7s to 5.6s — Distinct-Pie2389 · 2026-10-24
- AI's 'body' is a GPU box in theory, but inference is physically distributed in practice — jd_pressman · 2026-10-24