Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices
Yangyijian Liu, Hongyi Ye, Mingyang Li, Wu-jun Li
cs.DC, cs.AI, cs.LG
2026-07-11
ATSInfer schedules LLM inference at tensor granularity across CPU and GPU, combining static placement, load-aware dynamic transfer, and async pipelining to achieve up to 3.29x decode and 1.94x prefill speedup over llama.cpp on consumer hardware.
Running large language models locally on consumer devices is constrained by a fundamental mismatch: even quantized model weights routinely exceed GPU VRAM, forcing hybrid CPU-GPU execution. Existing systems schedule at layer or expert granularity, which has two compounding blind spots. First, tensors within the same layer differ substantially in compute intensity and memory access pattern, so the GPU acceleration benefit per byte of VRAM varies widely. Second, consumer hardware operates under dynamic load from background applications and thermal throttling, yet existing schedulers fix their tensor placement before inference.
ATSInfer addresses both limitations: it schedules at tensor granularity to exploit intra-layer heterogeneity and adjusts transfer decisions at runtime to track changing hardware conditions.
ATSInfer extends llama.cpp with approximately 15,000 lines of C++ and organizes its logic into three coordinated mechanisms.
Static tensor placement. Before inference, ATSInfer profiles each tensor's execution latency on both CPU and GPU and derives an empirical performance density (latency per byte of VRAM), which quantifies the acceleration gain per unit of GPU memory. Placement is formulated as a knapsack optimization: under a VRAM budget M, maximize the total latency reduction from GPU-resident tensors while penalizing backend switches between adjacent tensors. For dense models, a standard dynamic program solves this exactly; for MoE models, expert and non-expert tensors are partitioned first.
Load-aware dynamic transfer. Before each decode step, ATSInfer refreshes its estimates of CPU speed, GPU speed, PCIe bandwidth, and the computation-transfer overlap window from recent measurements. A second dynamic program determines which CPU-resident tensors should be temporarily promoted to the GPU. Only the exposed portion of weight transfer time — the fraction not hidden behind preceding computation — lands on the critical path. Re-scheduling is rate-limited when deviation is below 15% to avoid overhead.
Asynchronous CPU-GPU coordination. Two CUDA streams separate computation from weight transfer: activation transfers use an SM-driven copy kernel via Zero-Copy (accessing pinned host memory directly from GPU threads), while weight transfers use the dedicated hardware Copy Engine. Routing small latency-critical activation transfers through a parallel SM path prevents them from being blocked behind large weight transfers, maximizing overlap.
Experiments span a laptop (Intel i7-11800H + RTX 3060 6GB) and a desktop (Intel i7-11700 + RTX 4090 24GB), testing Qwen3-14B, Qwen3-30B-A3B, GLM-Z1-9B, and others against llama.cpp, vLLM, and KTransformers.
Compared with llama.cpp, ATSInfer achieves up to 1.94x prefill throughput and 3.29x decode throughput on the laptop; against vLLM, the decode speedup reaches 4.35x. GPU SM utilization during decode increases by roughly 70% on average. Under simulated dynamic load, the load-aware policy reduces TPOT degradation by 13% under 60% external CPU pressure and by 38% under 60% GPU+PCIe pressure.
Local LLM deployment has been bottlenecked not by model quality but by inference latency on resource-constrained hardware. ATSInfer demonstrates that principled scheduling can deliver close to 3x decode speedup on existing consumer devices without model changes. The empirical performance density metric and unified dynamic-programming formulation covering execution cost, switching overhead, and transfer overlap offer a concrete framework that future system designers can build on. The approach applies to both dense and MoE models and its tensor-level decisions are orthogonal to expert-caching strategies, leaving room for combination.
The evaluation is confined to two NVIDIA GPU platforms; behavior on AMD or Apple Silicon hardware is not assessed. ATSInfer does not integrate MoE-specific expert residency management, so systems like KTransformers may retain an edge when expert activation is highly concentrated. Re-scheduling overhead (16 ms total) is non-trivial relative to decode latency (40 ms per token for Qwen3-14B on RTX 3060). Static placement depends on offline profiling, adding initialization cost for new model-hardware combinations.