Open Pretraining Run Matches Llama 3.2 1B at ~90% Lower Cost per Token
jon_durbin · x · 2026-10-04
Developer jondurbin wrapped up his open pretraining "Kappa run" with strong preliminary results: at 9T tokens and a 576B checkpoint, the model beat Llama 3.2 1B on ARC-C, OBQA, TQA and more, while nearly matching it on ARC-E and SciQ — at roughly 90% cheaper per token.
Known issues include an underflow in the per-channel decay gate of one Gated DeltaNet-2 head. Next steps: increase model width and remove shared experts, since reasoning performance was lackluster — though the GDN2 issues make attribution uncertain.
Related event: Open Pretraining Matches Llama 3.2 1B at One-Tenth the Cost(3 posts)→
More from Research
- ENCODE preprint: neural nets predict super-additive GATA motif epistasis (r=0.55) — anshulkundaje · 2026-11-19
- New TLAPS-Bench blog: formally proving correctness of real-world system specs with TLA+ — tianyin_xu · 2026-11-19
- Stanford's Kundaje: ChIP-seq resolution limits motif-level deep learning analysis — anshulkundaje · 2026-11-19
- Mahina 4.0 LoRA Adapters Claim Fewer Hallucinations, Devs Seek Stress-Tests — External-Fee-8920 · 2026-11-19
- The biggest advantage of a top lab isn't money — it's saving you years — madeofAjala · 2026-11-19
- LEV: Looped decision model lifts 2.6B accuracy 63.2% to 76.2% with adaptive compute — jiqizhixin · 2026-11-19