Inside a pretraining run: MoE kernel limits forced 4k context, then mid-run kernel wins lifted MFU 20%→30%

dlwh · x · 2026-10-25

• classiclarryd shares internal details of a pretraining run: longer docs in the data meant launching at 16k context like MAI was intended, but the MoE kernel only supported 4k (high dropping), so the scaling ladder was built at 4k, knowing it would cease to apply once the new kernel landed.
• Remarkably, that and many other kernel improvements landed mid-run, raising MFU from 20% to 30%.
• The Uncheatable forecast was added mid-run without calibrating its asymptote or fitting to run data, so it's quite off; its main value was directionally estimating the impact of an earlier datamix change. dlwh notes Larry again has the actually useful details.

Original post →

More from Infra

• Pushing local AI video to the limit: MiniMax H3 running on a single RTX 3090 — DeliciousGorilla · 2026-10-25
• a16z: GPU rental rates defy depreciation, H200 up 143% YTD — elonmusk · 2026-10-25
• Qwen3.8 Flash Next IQ3_S runs locally on 16GB VRAM at 250 tok/s generation, 5,600 tok/s prefill — yehyakar · 2026-10-25
• 4x DGX Spark cluster recipe runs GLM-4.6-Flash with 1M context, 16 concurrent, fully local — AIFlow_ML · 2026-10-25
• $0.48 and 43 minutes: DeepSeek V4.1 builds an interactive map, igniting cost vs capability debate — teortaxesTex · 2026-10-25
• Do ISTA's 2-bit GSQ-RCO DeepSeek quants really match the base model? Community asks for KLD — Queasy_Back2588 · 2026-10-25

Infra channel →