fabric-lib: RDMA Point-to-Point Communication for LLM Systems
Nandor Licker, Kevin Hu, Vladimir Zaytsev, Lequn Chen
Perplexity AI
cs.DC
2025-11-01
fabric-lib unifies ConnectX and AWS EFA via unordered WriteImm at 400 Gbps. 1T-param RL weights land in 1.3s; MoE decode matches DeepEP on CX-7 and is first viable on EFA.
Modern LLM serving and training keep inventing traffic that collectives handle poorly. Disaggregated inference has to ship KvCache pages between prefiller and decoder machines. MoE has to scatter tokens to experts and gather them back. Async RL has to push freshly updated weights onto inference GPUs after every step.
NCCL and torch.distributed are built for static groups: fixed membership, a synchronized world, matching buffer shapes. Elastic prefiller and decoder pools fight the membership model. Sparse MoE routes get inflated into dense transfers. Send/Recv exist, but composing them into low-latency pipelines has not worked in practice.
RDMA Write has been the HPC answer for years. The blocker is the NIC zoo. NVIDIA ConnectX uses Reliable Connection with in-order delivery. AWS Elastic Fabric Adapter (EFA) uses a proprietary Scalable Reliable Datagram (SRD) that is reliable and unordered. DeepEP needs GPU-initiated RDMA (IBGDA), which only ConnectX provides. NVSHMEM falls over on EFA. Mooncake and NIXL, at the time of this work, had no EFA path or only a sketch of one. There was no portable P2P stack for LLM systems on mixed cloud hardware.
The shared subset across ConnectX and EFA is reliable, unordered delivery: ConnectX RC can ignore ordering, EFA SRD never had it. fabric-lib's TransferEngine is built on that subset.
The primitive is one-sided WriteImm plus ImmCounter. Each write can carry a 32-bit immediate. The receiver tallies immediates from completion queues and fires a callback once a target count is reached. No message-order assumption. Correctness rides on PCIe write ordering: the payload of WriteImm is posted before the immediate, and after the CPU sees the count it issues the next GPU-facing operation, which the PCIe switch orders after the NIC-to-GPU data writes.
The engine is Rust. One worker thread per GPU, pinned on the local NUMA node. Multi-NIC topologies are hidden: one ConnectX-7 already offers 400 Gbps; AWS p5 needs four 100 Gbps EFA NICs (two 200 Gbps on p5en) to match. Transfers are sharded and rotated across NICs. EFA goes through libfabric, ConnectX through libibverbs, with Send/Recv and WriteImm split onto two RC queue pairs so their completions do not steal work requests from each other. GPU progress is observed with a UVM watcher: a CUDA Graph kernel bumps a unified-memory word, the CPU polls it with GDRCopy, and the next layer's transfer starts. This is a host-proxy design, not GPU-initiated RDMA.
Three production paths:
Two clusters: 8x H200 with 2x 200 Gbps EFA per GPU, and 8x H100 with 400 Gbps ConnectX-7 per GPU.
| Op | EFA | ConnectX-7 |
| Single Write 256 KiB | 54 Gbps | 116 Gbps |
| Paged Write 64 KiB | 364 Gbps | 370 Gbps |
| Single Write 32 MiB | 336 Gbps | 378 Gbps |
Paged 64 KiB is a typical KvCache page; both NICs saturate. Single Write needs about 16 MiB to saturate, which is why 256 KiB MoE messages leave EFA underfed. TransferEngine is slightly ahead of NIXL v0.6.1.
KvCache on Qwen3-235B, H200 TP4, 32 kB pages (128 tokens). At 16K sequence length, TTFT goes from 929 ms (colocated) to 1042 ms (disaggregated). Per-layer transfer is 1.61 ms against 9.86 ms of compute, so the copy hides under the layer. The TTFT gap is mostly an extra decode of the last prompt token in their engine, not the RDMA. Rust UvmWatcher callbacks sit at 6.2 µs p50.
RL weight push for Kimi-K2 (1T params): 256 training GPUs in BF16 (FSDP/PP/EP = 16/2/8) to 128 inference GPUs in FP8 (EP=32). One-rank wall time is 1233 ms. The critical path is fulltensor at 518 ms and waiting on other ranks at 357 ms. Posting 1144 RDMA work requests costs 26 ms; only 42 ms of RDMA sits outside the overlap. DeepSeek-V3-671B and Qwen3-235B land in the 1.2-2 s band. Prior public numbers for similar sizes are 10-100 s. The intro's "100x" is a stretch against that range; 8-80x is the honest interval.
End-to-end MoE decode (DeepSeek-V3, MTP, EP=DP=64, tokens/s):
| Cluster | Kernel | batch=2 | batch=8 | batch=32 |
| H200 EFA | Ours | 66.8 | 56.5 | 32.0 |
| H200 EFA | pplx-kernels | 21.0 | 11.6 | 4.9 |
| H100 CX-7 | Ours | 78.4 | 67.7 | 36.1 |
| H100 CX-7 | DeepEP | 73.8 | 65.8 | 36.3 |
On EFA this is 3-6x the NVSHMEM-based pplx-kernels, and the first latency the authors call viable. On ConnectX-7 it matches or slightly beats DeepEP, even with a host proxy instead of IBGDA.
Kernel-level decode at EP64: dispatch is 190 µs on CX-7 vs DeepEP's 180 µs; combine is 311 vs 327. At EP16 and EP32 both sides win. At EP64 the proxy spends about 1 µs enqueueing each of 56 inter-node peers, which shows up on dispatch. EFA decode trails ConnectX by about 30%. Prefill favors DeepEP, especially combine, where a sender-side partial sum cuts RDMA bytes (and drops accumulation to bf16).
This is a P2P layer that already carries three production patterns across two NIC families, not a ConnectX-only kernel contest. Teams serving MoE, splitting prefill/decode, or doing async RL on AWS previously had almost nothing to call.
Keep collectives for tensor and data parallel. Use this for elastic membership, sparse routes, and one-sided weight push. The library is open-sourced in pplx-garden. The API does not name a NIC; RC-compatible cards such as eRDMA or Broadcom would reuse the ConnectX path without changing application code.
It is incremental systems work. The idea that pays rent is treating reliable-but-unordered delivery as the contract, then making multi-NIC aggregation and completion notification reusable.
GPU-initiated RDMA is still missing on most cloud SKUs (AWS p5/p5e, eRDMA) and only preliminary on p5en, so the host proxy is a concession. Wide NVLink domains such as GB200 NVL72 will move MoE off RDMA entirely; the MoE kernels here have a shorter horizon than KvCache and RL, where RDMA already hides under compute.
Prefill skips DeepEP's intra-node replica via NVLink and sender-side partial sum. The decode-oriented kernels do not chunk transfers, so receive buffers are sized for the worst case and the deployable model set shrinks. Every peer must have the same NIC count per GPU. Read and atomics were left out on latency grounds.
End-to-end numbers come from a custom PyTorch engine, not the DeepEP integrations in vLLM or SGLang. NIXL later grew a preliminary EFA backend, so the portability gap will narrow. The "100x" RL claim compares 1.2 s against a 10-100 s literature range; do not quote that factor as a single measurement.