Running Qwen3.5-0.8B on a Sub-$20 CPU with Under 512MiB Memory

buryhuang · reddit · 2026-08-12

A developer successfully ran the Qwen3.5-0.8B model on a 2017 Amlogic A113X chip (quad Cortex-A53, no NPU/GPU) priced at $10-20.

Performance: Scored 4/5 on a 5-case smoke test of ARC-Easy. On-device inference achieved Prefill at 2.92 tok/s and Steady decode at 1.82 tok/s.
Resource Usage: Peak RSS was only 490 MiB with zero swap, utilizing 334% of 4 CPU cores.
Technical Implementation: The author wrote a custom C runtime compiled into a single static binary, abandoning Python and llama.cpp. The core idea is to compile a pinned model for a specific CPU target ahead of time instead of using a general runtime.
Engineering Insight: Decode throughput is bounded by memory bandwidth divided by bytes per token, independent of GPUs. The author notes the run is still compute-bound even after a 4.42x kernel speedup, leaving headroom for further optimization. For comparison, the same model with a generic C runtime on an M3 Pro decodes at 2.39 tok/s, meaning this cheap chip reaches 75% of that performance.

The author argues that a lot of inference doesn't actually need a GPU, and existing deployed low-spec hardware (like smart home hubs) is fully capable of running models directly.

Original post →

Running Qwen3.5-0.8B on a Sub-$20 CPU with Under 512MiB Memory

More from Infra