zml/llmd inference server runs on NVIDIA, AMD, Intel, TPU and Metal with images as small as 142MB
siwu · reddit · 2026-11-07
zml released zml/llmd (llmd-20261106.0), a universal inference server claiming top-tier speed that runs the same engine across NVIDIA, AMD, Intel, TPU and Metal.
Features
- Continuous batching, prefix caching, speculative decoding (DFlash/DFlash2/DSpark), tool calling, automatic tensor and experts-parallel sharding.
- Native VFS with s3:// and hf:// support, streaming weights straight to GPU.
Model support: Qwen 2/3/3.5/3.6/3.8 series (dense and MoE), Gemma 3/4, LLaMa 2/3, Mistral 3 and Ministral, GLM 5.2/5.3, DeepSeek 4.1 Flash, LFM series, Xiaomi MiMo V2.6, and dozens more.
Image size advantage: CUDA 1.96 GB, ROCm 4.77 GB, TPU 272 MB, OneAPI 454 MB, Metal 142 MB (brew install) — far smaller than alternatives.
More from Infra
- Tesla targets chip fab inefficiency: wafers spend as little as 20% of time in useful processing — XFreeze · 2026-11-07
- Redditor runs Qwen Flash Next locally at 24 tok/s on 64GB MacBook via Strata SSD offloading — New-Pressure-6932 · 2026-11-07
- Reactor co-founder publishes 'The real-time manifesto' on serving real-time AI at scale — smtabatabaie · 2026-11-07
- Clairvoyance's new hivemind feature lets you share your GPU with anyone — draginol · 2026-11-07
- Sequoia's Maguire models SpaceX taking inference compute share as neoclouds struggle — shaunmmaguire · 2026-11-07
- Musk teases 'Starmind computer' details next year, promising it will 'blow people's minds' — ns123abc · 2026-11-07