CS student's Rust framework runs nanoGPT bit-for-bit identical to PyTorch at 2MB vs 580MB
Mental-Feature-2377 · reddit · 2026-10-17
A CS student built DenseCore, a Rust deep-learning framework with a PyTorch-like Python API. The standout: point import torch at it and karpathy's nanoGPT produces bit-for-bit identical results—same initial weights from manualseed, same loss, gradients, and weights across all 50 steps tested, even with dropout on.
The journey surfaced PyTorch internals: CPU attention isn't the textbook formula but a flash attention kernel that switches paths with dropout; adam rounds differently on tail elements of each thread's chunk due to compiler fusion; clipgradnorm secretly does reciprocal() maxnorm instead of division. Notably, PyTorch doesn't match itself across machines: same version, mac vs linux x86, only 1 of 26 ops gave identical bits—so DenseCore matches PyTorch only on Apple Silicon Macs.
It's 2MB vs PyTorch's 580MB. He's weighing directions: keep copying PyTorch, become a lightweight CPU drop-in for serverless/CI, or build a tool explaining why two PyTorch runs differ.
More from Research
- MathEconLib open-sourced: Axiom Math and top economists bring machine-checked proofs to economics in Lean 4 — soumitrashukla9 · 2026-10-23
- Ken Ono invites the community to build MathEcon on the new Lean 4 economics library — soumitrashukla9 · 2026-10-23
- IFM open-sources RL360: 8,192 concurrent sandboxes and ~1.3M rollouts for agentic RL — rupspace · 2026-10-23
- Science lifts expression of concern on 2024 AI debunking-conspiracies paper; conclusions unchanged — DG_Rand · 2026-10-23
- Embodied First Exam benchmarks coding agents on 1,090 tasks across 44 robots — ManlingLi_ · 2026-10-23
- Anima Anandkumar on neural operators: AI weather models beat supercomputers — AnimaAnandkumar · 2026-10-23