Engineer Uses Claude Code to Build Tiny Neural Net Hitting 10k+ tok/s on CPU

Engineer Gregory Diamos had a practical need: with no budget for large GPU clusters, his data processing pipeline required an ultra-tiny model capable of roughly 10k tokens/second on a single CPU core. Instead of hand-designing a network, he gave Anthropic's Claude Code a large token budget to autonomously explore "ridiculously small" neural networks, and shared several notable findings, prompting the community to reconsider the value of micro neural networks.

Confirmed

The goal of 10k tok/s CPU inference for data processing was set and carried out by Diamos; Claude Code autonomously produced valuable findings under a high token budget.
On an Intel Emerald Rapids (Xeon) CPU with AMX acceleration, packing active parameters into L2 cache yielded a generation speed of 14572 tok/s in a mostly bf16 configuration; single-core throughput was about 6616 tok/s, with emergent basic reasoning capabilities.
Training loss decreased monotonically.
Diamos noted that better weight compression could fit more active parameters and further boost performance.
Diamos explicitly cautioned that the results are experimental and not suitable as a production model.
On 09-08, @seanmcdonaldxyz extended the "super-compression" idea, demonstrating a mere 1.3KB script that expands into an entire T9 system, arguing micro networks deserve a redo; @HAXXVII relayed the experiment and urged the community to follow up.

Why it matters

This is a case of "using big models to build small ones": Claude Code autonomously designing networks under a high token budget showcases the potential of AI-assisted engineering in architecture exploration.
An ultra-small model packed into L2 cache achieving 10k+ tok/s on CPU, with basic reasoning emerging even on a single core, offers direct reference value for high-throughput CPU inference; the community has begun validating miniaturization and super-compression ideas, and related papers have sparked heated discussion.

2026-09-07 ~ 2026-09-08 · 8 related posts

Primary sources

14,572 tok/s on Intel Xeon: fitting active params into L2 cache with AMX — GregoryDiamos · 2026-09-08 01:37 (UTC+08:00)
Engineer uses Claude Code to build tiny CPU neural nets hitting ~6k tok/s per core — GregoryDiamos · 2026-09-07 17:59 (UTC+08:00)
Tiny neural nets hit emergent basic reasoning at 6,616 tok/s on a single Intel AMX core — GregoryDiamos · 2026-09-08 08:53 (UTC+08:00)

Developer uses Claude Code to build a tiny 10k tok/s CPU model, shares 3 findings — GregoryDiamos · 2026-09-07
Researcher uses Claude Code to build a tiny 10k tok/s CPU neural net, loss still descending — GregoryDiamos · 2026-09-07
Active params in L2 cache hit 14,572 tok/s on Intel Emerald Rapids with AMX — GregoryDiamos · 2026-09-07
1.3 kb Python script unfolds entire T9 autocomplete system — seanmcdonaldxyz · 2026-09-08
[source] 14,572 tok/s on Intel Xeon: fitting active params into L2 cache with AMX — GregoryDiamos · 2026-09-08
[source] Tiny neural nets hit emergent basic reasoning at 6,616 tok/s on a single Intel AMX core — GregoryDiamos · 2026-09-08

2 near-duplicate retellings: HAXXVII_ · GregoryDiamos