Running Qwen3.5-0.8B in pure Assembly: 6.51 tok/s on one CPU thread
suayptalha · reddit · 2026-11-08
Developer suayptalha implemented full inference of Qwen3.5-0.8B in pure Assembly, open-sourced on GitHub.
- On an AMD EPYC 7B12, single-thread generation hits 6.51 tokens/s vs 4.34 for the Transformers reference; all 32 generated tokens matched across runs.
- Tokenization, inference, and sampling are all in Assembly — no libc or Python at runtime (Python only builds tokenizer tables at compile time).
- The Assembly build reads BF16 weights, while the Transformers baseline loaded FP32 and used PyTorch fallback kernels; benchmark was 19 input tokens, greedy decoding, 2 runs.
Not an apples-to-apples comparison, but a striking demo of minimal-stack CPU LLM inference.
More from Infra
- Local LLM manager fix: concurrent requests from one session yield 2-3X speedup — draginol · 2026-11-09
- Datadog still leads observability, but its reputation is deeply split — thedealdirector · 2026-11-08
- Specialized Inference Chips: 3x Faster but Idle Half the Day Isn't 3x Cheaper — ai · 2026-11-08
- Baseten Loops goes GA with serverless SFT/RL training, per-token pricing — baseten · 2026-11-08
- Where do your agents run when the laptop closes? 4 hosting options compared — michaelTM_ai · 2026-11-08
- Google Donates Agent Substrate and gVisor to CNCF to Sandbox AI Agents at Scale — emmanuelvivier · 2026-11-08