A 5KB pure x86-64 assembly engine runs Gemma-2B at 4.6 tok/s on CPU
tom_tsai28 · reddit · 2026-10-04
Developer tomtsai28 shares PULSAR-ASM, a personal project: an inference engine for Gemma-2B written entirely in flat x86-64 assembly (FASM), exploring the minimal bare-metal footprint for running an autoregressive LLM.
Key facts:
- Total 5.2 KB of machine code (3.7 KB engine + 1.5 KB custom AVX2/F16C GEMM kernel)
- Pure AVX2 + F16C with a custom 4-thread SMP GEMM for prefill, sustaining 18.5 GB/s memory bandwidth on commodity DDR4-2400
- 4.5–4.7 tokens/s FP16 decoding on an older quad-core i5 desktop
- Zero C/C++ runtime, zero PyTorch; the Python harness only uses ctypes for VirtualAlloc and OS threads
Not meant to compete with llama.cpp — it's a first-principles exploration of how cleanly a modern Transformer maps to raw silicon, and a reference point for future micro-LLMs on constrained MCUs/DSPs. Code and architecture notes are open-sourced on GitHub.
More from Infra
- DeepMind researcher: global AI compute growing at least 10x per year — 141_1337 · 2026-11-20
- GPU shortage debate: companies with real AI demand are still massively compute starved — matt_slotnick · 2026-11-20
- Microsoft's Bing Web IQ Goes GA on Azure: Auto Orchestration Delivers Fresh Grounding for Agents — dejanseo · 2026-11-20
- exe.dev ships Rex integration, making remote VMs the default agentic dev workstation — davidcrawshaw · 2026-11-20
- Ex-Waymo exec Abe Murray: CUDA and ISAs won't stay moats, huge chip innovation incoming — IgorCarron · 2026-11-20
- MIT and NVIDIA's QDA uses agents to auto-design LLM quantization recipes, beating top public ones by up to 2.2 points — omarsar0 · 2026-11-20