Custom compiler runs Qwen3.5-0.8B BF16 on one CPU core with 109MB peak RSS via disk reads, seeking baselines

Maui-The-Magificent · reddit · 2026-11-23

A systems engineer shows a mid-way result from Cuddler, a model re-representation compiler: running Qwen3.5:0.8B BF16 inference via disk reads through a decoder built for graphics work, on a single CPU core with a static 256MB arena and only 109MB peak RSS.

The iterative decoder needs little data in memory, far below typical local inference footprints. The author admits a mechanical rather than runtime-level understanding of models and asks the community for RSS baselines from their disk-inference setups to judge his numbers, with weight re-representation as the next step. Screenshot shows per-turn checkpoint stats at 10-25 seconds per turn.

Original post →

More from Infra

Infra channel →