Custom compiler runs Qwen3.5-0.8B BF16 on one CPU core with 109MB peak RSS via disk reads, seeking baselines
Maui-The-Magificent · reddit · 2026-11-23
A systems engineer shows a mid-way result from Cuddler, a model re-representation compiler: running Qwen3.5:0.8B BF16 inference via disk reads through a decoder built for graphics work, on a single CPU core with a static 256MB arena and only 109MB peak RSS.
The iterative decoder needs little data in memory, far below typical local inference footprints. The author admits a mechanical rather than runtime-level understanding of models and asks the community for RSS baselines from their disk-inference setups to judge his numbers, with weight re-representation as the next step. Screenshot shows per-turn checkpoint stats at 10-25 seconds per turn.
More from Infra
- Fine-tuned Inkling 2 Mini routes tickets with early exit: median latency 136.8ms to 81.4ms at 0.07% accuracy cost — eltokh7 · 2026-11-23
- Splash 1.7.1 cuts Apple Silicon TTFT 23% via Neural Engine prefill, 2.9x faster shared-prompt chats — oss-jiiim · 2026-11-23
- From-scratch Metal 4 engine runs Inkling-2-Mini at 144 tok/s on M5 Ultra, 128k context in 32GB MacBook — DankpawsDev · 2026-11-23
- LFM3-Omni-2B runs locally on iPhone 17 Pro Max doing agentic web search with native speech generation — fuzhongkai · 2026-11-23
- Melt_Dem: Every Layer of Inference Optimization Will Collapse Into a Market — const_reborn · 2026-11-23
- Canary tokens in ~2M AI requests expose breaches at 3 providers and broken ZDR promises — ccerrato147 · 2026-11-23