TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure
Yuhan Zhou, Yuchu Luo, Hao Nie, Wangrunze Lv, Yu Zhou, Yibo Zhu, Daxin Jiang, Chenren Xu
cs.DC
2026-08-06
A standalone tensor lifecycle layer that matches specialized weight-loading, KV-cache, and checkpoint systems while enabling routing policies that cut median TTFT by up to 93.2%.
In LLM serving, tensors stopped being throwaway computation data. Model weights get distributed to newly launched instances during autoscaling, KV caches get reused across nodes, and RL post-training pushes updated weights from trainers to rollout workers. Each task grew its own optimized system: ServerlessLLM and InstantTensor for weight loading, Mooncake and LMCache for KV caches, ByteCheckpoint for checkpoints. Every one of them reimplements the same handful of operations, identify, place, move, transform, materialize, and welds those operations to a specific engine, network, or storage backend.
The silos cost twice. The same movement machinery gets rewritten per workload, and worse, policies that span components cannot be expressed: a router that jointly balances instance load and KV cache locality requires coordinated changes to the router, the KV system, and the execution engine.
The proposal, Tensor-as-a-Service (TaaS), is that tensor lifecycle management should be pulled out of computation logic into its own layer. TensorCast implements it: about 160K lines of C++ runtime plus a roughly 95K-line Python SDK, open-sourced, with integrations for vLLM and SGLang.
Four programming abstractions form the interface:
The runtime has three roles: workers (basic contributes memory and disk, gateway terminates caller connections, shard home owns partitions), instance nodes that host an engine plus an adaptor as the mechanism boundary, and a Global Store holding control-plane metadata in DuckDB, off the data path. Metadata handling splits by tensor kind: low-cardinality tensors such as weights keep replica locations in the Global Store; high-cardinality ones such as KV pages are sharded by HRW hashing, with the top-3 candidate workers competing for a lease that carries a monotonically increasing fencing token against split-brain. On the data plane, same-host GPUs share memory through CUDA IPC, cross-node transfers use RDMA when available and fall back to userspace mTCP with multipath.
Hardware: 8x H800 for weight loading, two nodes of 4x H800 for weight sync, a 200 Gbps RDMA cluster for KV, and four nodes of 2x H20 for routing.
| Scenario | Baseline | Result |
| Weight load, 30B, JuiceFS, cold | vLLM default / InstantTensor | 60.7x / 10.2x faster |
| End-to-end launch, 235B, JuiceFS, warm | vLLM default / InstantTensor | 228.6x / 40.7x faster |
| Weight sync in SGLang, no RDMA | default updater | 1.14x to 2.63x faster |
| KV reuse, 32B, TP=2, RDMA | Mooncake | 60% to 87.5% TTFT reduction, on par |
| Router, 256 medium sessions | load-aware + Mooncake | 93.2% lower median TTFT |
The 228.6x figure needs its precondition stated: warm start, weights already materialized in the pool, vLLM picks them up through CUDA IPC in under a second, and launch time is dominated by runtime initialization. The honest first-launch number is the cold one, 60.7x for the 30B model. Without RDMA, TensorCast clearly beats Mooncake on KV transfer, thanks to mTCP multipath aggregating bandwidth, which matters for cross-datacenter deployments without RDMA interconnects. The router experiment replays OpenHands trajectories over roughly 2,400 real Python repository tasks from SWE-Gym as multi-turn sessions; median TTFT drops 71.4%, 93.2%, and 70.4% under fast, medium, and slow inter-turn delays, with the smallest decline in cache hit rate.
For inference platform engineers, the value is not any single speedup but writing the tensor movement machinery once and reusing it across four scenarios that today are four codebases, with per-scenario performance holding up. The bigger unlock is cross-component policy: a router that jointly considers load and KV locality is a few dozen lines of caller code here, versus coordinated changes to the engine, KV backend, and scheduler in siloed stacks, with the engine untouched. The authors are direct about positioning: they do not claim a general abstraction beats every specialized implementation; the claim is composability at no performance cost.
Stated by the authors: decoupling does not eliminate engine-specific integration, each engine still needs an instance adaptor; training frameworks (Megatron-LM, DeepSpeed) are future work; skipping ACID means partially executed plans leave residual state.
Worth questioning after a close read: all four evaluations ran on the authors' own Stepfun and PKU testbeds with no third-party reproduction; the router is compared against three policies in the SGLang gateway, not against newer joint schedulers (the cited DualMap is still under review); 228.6x assumes pre-materialized weights, while a production first launch lands in the cold regime; the weight-loading baseline set includes InstantTensor but leaves out serverless cold-start systems such as ServerlessLLM.