Open pretraining run matches Llama 3.2 1B on ARC-C at ~10% of the cost, author details 4 pitfalls
jon_durbin · x · 2026-10-03
jondurbin shared preliminary results and a post-mortem of the Kappa pretraining run:
Results: At the 576B-token checkpoint, the model beat Llama 3.2 1B (trained on 9T tokens) on ARC-C, OBQA, TQA and others, and nearly matched it on ARC-E, SciQ, etc. — at roughly 90% lower cost per token.
Issues encountered:
- A per-channel decay gate in one Gated DeltaNet-2 head underflowed, wiping state; output RMSNorm then amplified near-zero outputs 1000x into fleet-wide grad spikes;
- Very slow/late nodes had their evidence discarded; better tolerance for stale data and partial updates is needed;
- Samples were packed without masking, suboptimal for pretraining;
- GDN2 in the first and last layers may be suboptimal vs. SWA, though evidence is unclear.
He calls it "a solid preliminary win" and plans fixes.
Related event: Open Pretraining Matches Llama 3.2 1B at One-Tenth the Cost(3 posts)→
More from Infra
- AMD Loom boosts Qwen3.5-4B prefill up to 2.6x on Radeon 890M laptops — jfowers_amd · 2026-11-19
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- AWS Lambda Adds GPU Support via Managed Instances for Inference and Media Workloads — DavidWells · 2026-11-19
- Nvidia Allegedly Made GPUs Artificially Scarce to Keep Prices Sky-High — GaryMarcus · 2026-11-19
- AWS adds private CA support to Bedrock AgentCore Identity for connecting agents to self-hosted IdPs — AWS ML Blog · 2026-11-19
- Leak Hints at Classified 1M-GPU Training Cluster in China That Skips HBM — teortaxesTex · 2026-11-19