Dev builds paged LLM inference engine for Apple Silicon, cuts attention memory 128x
Accomplished_Row1433 · reddit · 2026-09-20
The author open-sourced PagedServe, an experimental LLM inference runtime for Apple Silicon tackling KV-cache fragmentation: fixed-size physical KV blocks, logical-to-physical block tables, direct blockwise attention with online softmax, copy-on-write prefix sharing, continuous batching, and an experimental Metal decode kernel.
Benchmark results:
- Temporary attention memory at 2048-token context drops from 12,288 KB to 96 KB per layer (128x);
- TinyLlama throughput on M3 Pro rises from 3.78 to 13.2 tok/s.
Caveat: the Metal kernel is currently 2.43x slower than the PyTorch blockwise path; profiling suggests the bottleneck is Python recreating Metal buffers and encoding a command buffer on every dispatch, not the attention math itself. The full runtime, benchmarks, and 283 tests are open-sourced, and the author seeks feedback on persistent Metal buffers and command-buffer reuse.
Repo: github.com/vermasarthak/pagedserve
More from coding & agent
- Da7em Bench: independent AI benchmark scores models on 200 real client tasks across 12 areas — airesearch12 · 2026-09-20
- Using Codex + GPT-6 Astra to plan art installations: rebuild the wall in Blender, skip recalculation — perilli · 2026-09-20
- What does an AI engineer's day actually look like? A learner asks Reddit — tech_kie · 2026-09-20
- Open-source local pipeline generates full radio dramas from one click — fflluuxxuuss · 2026-09-20
- OpenClaw Hits 390K GitHub Stars as Open-Source Computer-Use Agents Proliferate — Aiden_Tech_Ai · 2026-09-20
- AI Capex vs AI Opex: Stop Paying Agents to Re-solve the Same Problem — njyx · 2026-09-20