Pinterest Spent 3 Months Finding a GPU Crash Bug Caused by an Unused Container
arpit_bhayani · x · 2026-08-04
Pinterest spent three months chasing a network bug that caused their GPU training jobs to crash, only to find it was caused by a container they weren't even using.
The Incident
- Training jobs on k8s kept crashing with intermittent network loss.
- System logs pointed to AWS network driver resets, a self-healing mechanism that kicks in when a network thread lacks CPU time for 5 seconds.
Debugging Process
- Usual fixes like huge pages, jemalloc, and CPU affinity failed.
- Profiling revealed Kubelet spiking to 6.5% CPU (from its usual 1%) before crashes.
- The kernel was tracking 70,000 memory cgroups, with only 240 in use.
Root Cause & Fix
- Their GPU machines used the AWS Deep Learning AMI, which auto-starts an Amazon ECS agent. Since they ran Kubernetes, the agent had no cluster to join, crashing and silently leaking zombie memcgs.
- The Fix: Disable the ECS agent systemd unit in the base image.
Takeaway: Base OS images carry unexpected baggage. Auditing active systemd units is crucial.
Related event: Pinterest Finds Unused Container Caused GPU Crashes(2 posts)→
More from Infra
- MiniMax H3 Acceleration Benchmark: TE-Speed Delivers up to 1.785x Speedup — Commercial_Board9219 · 2026-08-04
- Cloudflare Launches CI SDK with AI Self-Healing Code Fixes — dinasaur_404 · 2026-08-04
- Gavin Baker Reveals SSI to Launch Model in August, Discusses AI Infra & GPU Prices — zephyr_z9 · 2026-08-04
- AI Trade Enters Stock-Picker Phase as Compute Supply Defies Narrative — tengyanAI · 2026-08-04
- ClickHouse Cloud Rebuilds Autoscaling Orchestration for Near Real-Time Reactivity — mgill25 · 2026-08-04
- 65-byte Malicious File Crashes llama.cpp: Open-source Library 'modelvet' Hardens Model Parsing — tetsuoai · 2026-08-04