NVIDIA boosts local agent serving: 1.9x llama.cpp on RTX 5090, 1.2x vLLM on Blackwell
vllm_project · x · 2026-09-05
The vLLM project highlights NVIDIA RTX Spark's new local-agent optimizations:
- Up to 1.9x llama.cpp throughput on GeForce RTX 5090
- 1.2x vLLM performance on RTX PRO 6000 Blackwell
- Up to 1.4x on a two-system DGX Spark cluster
The optimizations target the local serving path for agents, with weights available on Hugging Face.
More from Infra
- Corporate token spending growth collapses from 42% to 0.2% as hyperscaler ROIC falls to 20% — SumitGup · 2026-10-21
- Qualcomm plans worldwide performance-based layoffs as it chases $5B AI data center revenue — rohanpaul_ai · 2026-10-21
- vLLM at PyTorchCon 2026: KV cache and kernel optimizations across TPU, AMD, and NVIDIA — PyTorch · 2026-10-21
- antirez to port DeepSeek Infer kernels to DGX Spark's SM121 for DwarfStar — antirez · 2026-10-21
- Ternary-Bonsai-2-9B: 2-bit ternary vision-language model trends on Hugging Face for on-device use — prism-ml · 2026-10-21
- General Compute x Positron: GPUs Handle Prefill, Memory-First Chips Tackle Decode-Bound LLM Inference — cneuralnetwork · 2026-10-21