Dev builds open RK3588 NPU runtime, runs Qwen3.5-4B ~2x faster than vendor SDK
Flashy_Squirrel4745 · reddit · 2026-10-21
Developer happyme531 released openrknpu2lminfer, an open-source LLM inference engine for the Rockchip RK3588 NPU that works without Rockchip's proprietary userspace libraries — the host side is written in Rust and NPU operators in C++, fully hackable.
Performance vs RKLLM 1.3.1 (Qwen3.5-4B):
- Prefill, 512-token prompt: 147.83 vs 58.35 tok/s (2.5x)
- Prefill, 3,800-token prompt: 139.92 vs 54.03 tok/s
- Decode after 512 tokens: 10.61 vs 5.54 tok/s (2x)
Features: terminal chat plus an OpenAI-compatible API with streaming, tool calling and prefix caching; Qwen3.5-4B image input; validated checkpoints for Qwen3-4B, Qwen3.5-4B, Granite-4.2-3B and MiniCPM5-2B; prebuilt ARM64 binaries.
Planned next: DFlash/DSpark speculative decoding and MiniCPM-V 4.7 with expert prefetching. Repo: github.com/happyme531/openrknpu2lminfer
More from coding & agent
- Ethan Mollick's new book charts 30 months of AI progress, from cyborg to conjurer work — rseroter · 2026-10-23
- OpenJet claims Qwen3.8-27B coding runs 1.9x faster than llama.cpp on one RTX 3090 — JeremyCMorgan · 2026-10-23
- Jensen Huang details Vera Rubin agentic stack: NVL72, MGX, CMX and RTX — rohanpaul_ai · 2026-10-23
- Building an autonomous SEO agent in Hermes with three skills: programmatic pages, Keywords Everywhere, DataForSEO — EXM7777 · 2026-10-23
- Building a sports betting AI agent with onchain execution and human approval by default: architecture and lessons — Dry-Comfortable-2678 · 2026-10-23
- Capy 0.5.0 lets Claude Pro/Max subscribers reuse their plan quota, adds OIDC cloud access — garrytan · 2026-10-23