Andrej Karpathy releases llm.c: Train LLMs in raw C/CUDA
goyalshaliniuk · x · 2026-08-19
Andrej Karpathy has released llm.c, a project designed to train Large Language Models using simple, readable pure C/CUDA code without the need for 245MB of PyTorch or 107MB of cPython dependencies.
- Core Focus: Currently focuses on pretraining, specifically reproducing the GPT-2 and GPT-3 miniseries, alongside a parallel PyTorch reference implementation.
- Performance: The current mainline code in traingpt2.cu is approximately 7% faster than PyTorch Nightly.
- Code Structure: In addition to the CUDA implementation, it includes a clean, 1,000-line CPU fp32 reference implementation in a single file (traingpt2.c).
- Scope: The repo intends to maintain only C and CUDA code, avoiding ports to other languages.
More from Infra
- Anthropic's Multi-Level Monitoring for Astra Inference Revealed — AccBalanced · 2026-08-19
- Brex Report: Infrastructure Wins Over Apps in AI Hype — simonguozirui · 2026-08-19
- Alchemy Author: Not Just for Complex Projects — Simplest Way to Build Any Infra — samgoodwin89 · 2026-08-19
- Test: DeepSeek Harness achieves 99% cache hit rate with GLM and Kimi — sandyyevans · 2026-08-19
- 51WORLD Launches Embodied Data Infrastructure, Boosting Efficiency 10x — 量子位 · 2026-08-19
- AMD posts async RL walkthrough on MI355X and benchmarks vs B300 — AnushElangovan · 2026-08-19