Reverse-engineered NPU engine format to run GGUFs 1.5× faster than vendor runtime
woolcoxm · reddit · 2026-08-28
Author reverse-engineered the engine format of the Axera AX8850 NPU, discovering that int8 weights are stored as two nibble planes. They implemented a method to patch GGUF weights directly into precompiled engines at load time, eliminating the need for model conversion. The project also fixed a supposedly broken batched-prefill path, achieving 1.5× faster decode speed (24.5 t/s) and 716 t/s prompt processing on a Raspberry Pi 5 compared to the vendor's runtime. The analysis reveals the NPU's performance is primarily memory-bandwidth bound.
More from Infra
- Tabularis: Open-Source SQL Workspace with Built-in MCP Server for AI Agents — tom_doerr · 2026-08-28
- Thomson Reuters builds in-house AI model on Alibaba's Qwen for $40M — schwarzjn_ · 2026-08-28
- Ex-Nvidia engineer on company culture and inference infrastructure — Scobleizer · 2026-08-28
- Why Nvidia Won't Build Consumer GPUs: Fab Space and Token Demand — teortaxesTex · 2026-08-28
- MacBooks with 4G and no sleep on lid close would add billions to GDP via agents — gabriel1 · 2026-08-28
- Serverless V8 Isolates Hailed as Future for Running Coding Agents — tobowers · 2026-08-28