Reverse-Engineering RK3588 NPU: Open Compiler Runs GPT-2 at 36 tok/s
one_does_not_just · reddit · 2026-08-20
The author shares a year-long effort to reverse-engineer the RK3588 NPU. By cracking the register format and building an open-source compiler + runtime, they achieved model execution directly from PyTorch, ONNX, and JAX without the vendor SDK.
Results:
- Supports GPT-2 and SigLIP on RK3588 NPU.
- GPT-2 inference speed reaches 36 tok/s.
- A detailed technical blog and report are provided, covering everything from memory hacking to compiler architecture.
More from Infra
- Large data centers employ only ~50-150 permanent on-site staff, Microsoft says — brezina · 2026-10-16
- Prediction: Opus 5.5-level models on laptops by 2027, backed by Qwen 27B data — cephaloform · 2026-10-16
- Oracle down 12% in a week, nearly 50% off June high as OpenAI exposure spooks market — alvelda · 2026-10-16
- US data center moratoriums top 400 — an AI bull explains why they're actually good for AI — kevinsxu · 2026-10-16
- Microsoft Research unveils Magnetar, the fastest zkVM at 10-39x speedups and post-quantum security — jedisct1 · 2026-10-16
- Bonsai 2 2B hits 120 tok/s on iPhone 18 Pro, outpaces Qwen 3.5 2B at half the memory — cephaloform · 2026-10-16