jitllm moves a running LLM's layers between GPUs mid-reply in 8ms with zero recompute
web3samy · reddit · 2026-10-14
Inspired by OS and JIT compiler design, a developer built jitllm, an inference engine that treats model layers like schedulable processes: the paged KV cache moves with the layers, so no re-prefill is needed.
Benchmarks:
- On 2×T4, Llama-3.2-3B Q4KM migrated from GPU 0 to GPU 1 mid-reply in 8.1ms with 0 tokens recomputed; the reverse move took 8.4ms
- On 2×V100 with a 1.8k-token chat, migration took 20-21ms
- On an RTX 3050 Ti laptop, the model was split across CPU, CUDA and Vulkan (iGPU) and rearranged while generating, in 91ms
The author argues JIT-generated kernels bring every optimization to every machine, and that paging plus relocation turns placement into scheduling — enabling models larger than memory, mismatched GPUs, and workloads that shift as conversations grow. Code is open-sourced on GitHub.
More from Infra
- Runner trains LoRA directly on quantized GGUFs — no PyTorch, no FP16 base needed — ZenZombie117 · 2026-10-14
- The AI race belongs to whoever gets enough compute online — Meta and xAI prove it — haider1 · 2026-10-14
- GPUs are going on-prem: owning beats blended cloud at 33% utilization and neoclouds at 66% — AccBalanced · 2026-10-14
- Koboldcpp v1.123 adds support for Qwen Image 2.1 — Fcking_Chuck · 2026-10-14
- Inference Boom Runs Blind: H100 Hours Price from $2.20 to $8.00 with No Utilization Data — AccBalanced · 2026-10-14
- Hugging Face launches Kernel Hub to distribute compute kernels as first-class artifacts — ariG23498 · 2026-10-14