NVIDIA quietly open-sources Flare, a Rust-based local inference server for Blackwell GPUs
AIFlow_ML · x · 2026-10-21
NVIDIA quietly released Flare, an Apache-2.0 local inference server for SM120/SM121 GPUs (RTX PRO 6000 Blackwell workstations and DGX Spark), with almost no announcement and only 22 GitHub stars. Built for long conversations, coding agents, and repeated tool use: PyTorch + FlashInfer at build time, a Python-free Rust runtime at serve time, CUDA graphs captured before serving, FP8 KV cache by default, and recipes for Qwen Flash, GLM, and DeepSeek.
More from Infra
- vLLM's specialized decode path hits 829 P90 TPS per user on four B300 GPUs, 26% faster — vllm_project · 2026-10-21
- vLLM says it can reproduce DeepSeek's inference performance, yielding $39/GPU/hr revenue — vllm_project · 2026-10-21
- vLLM matches DeepSeek's DeepInfer decode performance within 6%, generating $39/GPU/hr at off-peak pricing — vllm_project · 2026-10-21
- AMD brings EAGLE-4 speculative decoding to Instinct MI355X day 0, 1.6x faster than built-in MTP — hongyangzh · 2026-10-21
- TokenGate: learned token pruning speeds up LLM prefill up to 11.3x — Liana Patel · 2026-10-21
- Modal revives multiplayer game Curvytron to demo its new Sticky Sessions routing — charles_irl · 2026-10-21