Tiny neural nets hit emergent basic reasoning at 6,616 tok/s on a single Intel AMX core
GregoryDiamos · x · 2026-09-08
Greg Diamos set out to solve an engineering problem, not write a paper: with no GPU cluster budget, he needed a model doing simple extraction and classification at 10k tokens/sec per CPU core. He gave Claude Code a pile of tokens to build one, and got three unexpected discoveries plus a paper and model.
Key constraints and findings:
- Every run is pinned to one physical core (Intel Xeon Silver 4514Y, OMPNUMTHREADS=1) — a roofline exercise, not a stunt
- AMX is worth 9x over AVX-512 fp32 at the model's shapes: best single bf16 GEMM at 2,231 GF/s, bf16 at 1024³ reaching 976–1,074 GF/s vs 121 GF/s for fp32 without AMX
- The punchline: if active parameters are small enough that hot weights fit in the 2 MiB L2 per core, 10k+ tokens/sec generation on one core is just arithmetic
The original post drew 120K views and 87 replies in a day; the writeup covers what the community pushed back on and where critics were right.
More from Infra
- Sail equivalence proof and riscv-coq cross-checks uncover real RISC-V spec bugs — xennygrimmato_ · 2026-10-27
- xAI's Colossus 1 hits ~14.6 exaflops FP64, ~5x Top500 leader — cis_female · 2026-10-27
- Lambda runs 19 B200 nodes in a 16-node power budget, boosting throughput 24% — TheZachMueller · 2026-10-27
- Podcast: Micron's HBM roadmap, Meta's AI infra, and NVIDIA's supply chain lock-in — BenBajarin · 2026-10-27
- SGLang MegaKernel + PD Pushes the TCO-Latency Frontier on GB300, SemiAnalysis Benchmarks Show — AccBalanced · 2026-10-27
- Scaling self-hosted Convex: compiling TypeScript Convex functions to WASM — leventov · 2026-10-27