Cache-busting bug masked a 24x GPU gap: 11 confident falsehoods, none from the model
colinmcnamara · x · 2026-10-13
Engineer Colin McNamara recounts 11 times his stack confidently told him something false — and not once was it the model. The last three were his own tooling: his cache-busting mechanism was itself cached, corrupting a laptop vs NVIDIA GH200 benchmark. Cold prefill was actually 169.6s on the laptop vs 7.1s on the GH200 — a 24x gap the opposite direction — but the cached results made it look like near parity. Lesson: in performance measurement, the assumption you "already fixed" deserves the most suspicion.
Related event: Engineer Finds 11 Confident Errors All Came From His Toolchain, Not the LLM(2 posts)→
More from Infra
- Next-gen laptops need a 35B MoE with ~9B active params, says dev — nothing above 3B exists — draginol · 2026-10-13
- Applied Materials: Agentic AI pods consume 2x-4x more wafers than Gen AI pods — BenBajarin · 2026-10-13
- AWS details fused MoE kernel engineering on Hopper, yielding 2-4% throughput gains — AWS ML Blog · 2026-10-13
- AMAT declines WFE forecast; analyst's own chart shows WFE going "to the moon" — BenBajarin · 2026-10-13
- Applied Materials CEO: AI drives WFE growth, yield innovation adds 10ppts margin by CY'30 — BenBajarin · 2026-10-13
- Bain: AI must earn $6T a year by 2031 to justify data centers — products max out at $1.8T — SumitGup · 2026-10-13